AI & Computer Vision, lesson 43 of 52
Bias, Fairness & the Model Card
Introduction
Your detector scores 85% overall. But is that 95% in sunshine and 40% under floodlights? 95% for the home kit and 50% for the away kit? An average can hide a group your system treats badly. Today you'll find out where bias comes from, measure your system situation by situation, and write a model card: an honest label on the box.
By the end of this lesson you'll be able to:
- Explain bias as a systematic difference in how well a system works for different groups or situations
- Name five places bias can enter: data, labels, measurement, deployment, feedback
- Evaluate a model by slices and compare each slice with the overall score
- Audit a dataset by class and session
- Write a model card with honest limitations
- Explain why we do not infer personal attributes about people from video
Working principle
A model is not neutral
A model learns from the examples it is given. If those examples are mostly one kind of situation, the model will work well there and badly elsewhere. Bias here means a systematic difference in how well the system works for different groups or situations. It is usually not anyone's intention. It comes from the data, the labels, the way we measure, and where we use it.
Where bias comes from
| Source | Example in our project |
|---|---|
| Data | All videos filmed on a sunny morning on one pitch |
| Labels | The person labelling only marked the ball when it was clearly visible, so blurry balls were never learned |
| Measurement | We measured success only on frames where the ball was easy |
| Deployment | Built for a school pitch with a tripod, used at a muddy tournament with a handheld phone |
| Feedback | The reel shows only the goals the system finds, so we only ever study those |
Averages hide groups
An overall score is one number for everyone. Splitting the test into slices shows who is served well and who is not. In Cell 1 the average was 82%, but one situation got 42%. The habit is simple: never report a score without the worst group beside it.
People are different from balls
Our test slices were about light and kit. When a system finds or analyses people, the stakes are higher. Real research has found that some commercial face-analysis systems made many more errors for darker-skinned women than for lighter-skinned men (the "Gender Shades" study, 2018). Nobody planned that; the training data and the testing data did not include enough of everyone.
For this course: do not build features that guess a person's gender, age, skin colour, religion or ability from a video. You do not need them to count goals. If you ever test fairness across groups of people, it needs consent, care and a teacher's guidance, because the labels themselves are personal data about children.
Fairness has no single formula
People disagree about what "fair" means. Equal accuracy for everyone? Equal chances? Equal costs of mistakes? Different definitions can conflict. So fairness is not only a number; it is a conversation with the people affected. What we can do is: look, measure, write it down, and fix what we can.
What a model card does
A model card (from a 2019 paper by Mitchell and colleagues) is a one-page honest description:
| Section | Question it answers |
|---|---|
| What it is for | What was it built to do? |
| What it is NOT for | What should nobody use it for? |
| Data | What did it learn from? Who or what is missing? |
| How well it works | Overall and by group |
| Limitations | Where does it fail? |
| People and privacy | What happens to anyone's data? |
| Contact | Who do I tell if it fails me? |
Check the data first
Many fairness problems are visible before any training. Count your data by class and by session (Cell 2). Look at the pictures. Ask who is not in them.
Fixing, and not fixing
You can often improve a weak group: collect more of that situation (lesson 29), augment (lesson 13), change the camera, change the threshold, or add a warning light (lesson 36). Sometimes you cannot. Then the honest thing is to write the limit down and to not use the system where it fails. A model card that says "do not use in the shade" is a better product than one that hides it.
Data about children (again)
India's Digital Personal Data Protection Act applies stricter rules to children's data. lesson 14 covers the dates. In practice: collect less, keep it for a shorter time, share it with fewer people, and ask for consent. A model card is a good place to say how you did that.
What you'll need
- Your detector and test results from Units 5 to 9
- Your
cv_courseenvironment
Build it
No real model yet? These cells test the lesson 36 colour ball finder on five groups of scenes. Copy scene.py, detector.py and degrade.py from lesson 36.
Cell 1: test by group, not just overall
"""Fairness starts with one question: does it work equally well for every group?
We test the lesson 36 ball finder on groups of scenes, and compare group by group."""
import cv2
import numpy as np
from detector import find_ball
from scene import ball_path, render
def render_variant(i, pos, variant, rng):
img = render(i, pos, rng)
x, y = int(pos[0]), int(pos[1])
if variant == "small_ball": # a distant camera: the ball is only a few pixels wide
img = render(i, (-50, -50), rng) # draw the scene without the ball ...
cv2.circle(img, (x, y), SMALL_R, (0, 200, 255), -1) # ... and add a tiny one
elif variant == "stand_shadow": # the shadow of a stand covers the left third of the pitch
img[:, :220] = (img[:, :220] * 0.5).astype(np.uint8)
elif variant == "yellow_kit": # a player in a ball-coloured bib runs through the picture now and then
if (i // 20) % 3 == 0:
cv2.circle(img, (min(x + 40, 630), min(y + 25, 350)), 16, (0, 200, 255), -1)
elif variant == "muddy_ball": # a duller, browner ball
cv2.circle(img, (x, y), 9, (40, 110, 150), -1)
return img
SMALL_R = 4 # ball radius in pixels for the "small_ball" group
VARIANTS = ["normal", "small_ball", "stand_shadow", "yellow_kit", "muddy_ball"]
def evaluate_groups(n_frames=120, tolerance=6.0, seed=0):
"""Returns {group: {'n':..., 'found':..., 'rate':...}} where 'found' = ball detected within `tolerance` px of the truth."""
rng = np.random.default_rng(seed)
path = ball_path(300)[:n_frames]
results = {}
for v in VARIANTS:
found = 0
for i, pos in enumerate(path):
frame = render_variant(i, pos, v, rng)
b = find_ball(frame)
if b and np.hypot(b["x"] - pos[0], b["y"] - pos[1]) <= tolerance:
found += 1
results[v] = {"n": len(path), "found": found, "rate": found / len(path)}
return results
def gap_report(results, max_gap=0.10):
"""Overall rate versus every group. Flags groups worse than the overall rate by more than max_gap."""
n = sum(r["n"] for r in results.values())
overall = sum(r["found"] for r in results.values()) / n
worst = min(results, key=lambda g: results[g]["rate"])
flagged = [g for g, r in results.items() if overall - r["rate"] > max_gap]
return {"overall": overall, "worst_group": worst, "worst_rate": results[worst]["rate"], "flagged": flagged}normal 120 of 120 100%
small_ball 120 of 120 100%
stand_shadow 50 of 120 42%
yellow_kit 80 of 120 67%
muddy_ball 120 of 120 100%
overall 82%
flagged: ['stand_shadow', 'yellow_kit']The overall number was 82%. It looks fine. But in the shadow of a stand the finder worked in only 42% of the frames, and when someone wore a ball-coloured bib it worked in 67%. An average can hide a group that is badly served. In a real club the "groups" might be: morning or evening light, grass or artificial turf, the near half of the pitch or the far half, different kit colours, boys' matches or girls' matches, new balls or old ones.
Cell 2: who is missing from our data?
If every image comes from one session, the model has only ever seen one day. Use the lesson 11 file names <class>_<session>_<n>.jpg.
"""Who or what is missing from our data? Count images by class and by session, from the lesson 11 file names: <class>_<session>_<n>.jpg"""
import os
from collections import Counter
def parse(name):
stem = os.path.splitext(os.path.basename(name))[0]
parts = stem.rsplit("_", 2)
return (parts[0], parts[1]) if len(parts) == 3 else (None, None)
def audit(files, min_share=0.10):
by_class, by_session, bad = Counter(), Counter(), []
for f in files:
c, s = parse(f)
if c is None:
bad.append(f)
else:
by_class[c] += 1
by_session[s] += 1
total = sum(by_class.values())
notes = []
for name, counter, what in (("class", by_class, "class"), ("session", by_session, "session")):
for k, n in counter.items():
if total and n / total < min_share:
notes.append(f"only {n / total:.0%} of the data is {what} '{k}': is it enough?")
if len(counter) == 1 and total:
notes.append(f"all data is from ONE {what} ('{next(iter(counter))}'): the model has never seen anything else")
if bad:
notes.append(f"{len(bad)} file names do not follow the pattern (e.g. {bad[0]})")
return {"total": total, "by_class": dict(by_class), "by_session": dict(by_session), "notes": notes}
if __name__ == "__main__":
import sys
folder = sys.argv[1] if len(sys.argv) > 1 else "."
r = audit(os.listdir(folder))
print(r["total"], "images")
print("by class: ", r["by_class"])
print("by session:", r["by_session"])
for n in r["notes"]:
print("NOTE:", n)Run python data_audit.py folder_with_your_images. A "NOTE" is a question, not a verdict: is 5% enough for the goal class?
Cell 3: write the model card
A model card is one page that says honestly what a model is for, what it is not for, how it was tested, and where it fails. A lint function checks that the card is complete and that the worst group is named in the limitations. It cannot make you honest, but it makes hiding harder.
"""A model card: one honest page that says what the model is for, how it was tested, and where it fails.
Idea from 'Model Cards for Model Reporting' (Mitchell et al., 2019)."""
REQUIRED = ["name", "version", "date", "authors", "purpose", "out_of_scope", "data", "overall", "slices",
"limitations", "people_and_privacy", "contact"]
def lint(card):
"""Return a list of problems. An empty list means the card is complete and honest enough to publish."""
problems = []
for key in REQUIRED:
v = card.get(key)
if v in (None, "", [], {}):
problems.append(f"missing: {key}")
elif isinstance(v, str) and any(w in v.lower() for w in ("todo", "tbd", "fill in", "lorem")):
problems.append(f"placeholder text in: {key}")
sl = card.get("slices") or {}
if sl:
rates = [r["rate"] for r in sl.values()]
if len(sl) < 3:
problems.append("test at least 3 groups of situations")
overall = card.get("overall", {}).get("rate")
if overall is not None and rates and (max(rates) - min(rates)) > 0.10 and not card.get("limitations"):
problems.append("groups differ a lot but limitations is empty")
worst = min(sl, key=lambda g: sl[g]["rate"]) if sl else None
lim = (card.get("limitations") or "").lower()
if worst and sl[worst]["rate"] < (overall or 1) - 0.10 and worst.replace("_", " ") not in lim and worst not in lim:
problems.append(f"the worst group ({worst}) is not mentioned in limitations")
if not card.get("people_and_privacy"):
problems.append("say what happens to people's data")
return problems
def render(card):
sl = card["slices"]
rows = "\n".join(f"| {g.replace('_', ' ')} | {r['found']} of {r['n']} | {r['rate']:.0%} |" for g, r in sl.items())
return f"""# Model card: {card['name']}
Version {card['version']} · {card['date']} · {card['authors']}
## What it is for
{card['purpose']}
## What it is NOT for
{card['out_of_scope']}
## Data
{card['data']}
## How well it works
Overall: **{card['overall']['rate']:.0%}** ({card['overall']['found']} of {card['overall']['n']} test frames, ball found within {card['overall'].get('tolerance_px', '?')} px).
**By situation** (the overall number hides these):
| Situation | Found | Rate |
|---|---|---|
{rows}
## Known limitations
{card['limitations']}
## People and privacy
{card['people_and_privacy']}
## Contact
{card['contact']}
"""Cell 4: tests
from model_card import lint, render
from slices import evaluate_groups, gap_report
bad = 0
def check(name, ok, extra=""):
global bad; bad += not ok; print("PASS" if ok else "FAIL", name, extra)
res = evaluate_groups()
rep = gap_report(res)
check("overall looks decent", 0.75 < rep["overall"] < 0.9, f"{rep['overall']:.0%}")
check("but a group is far worse", rep["worst_rate"] < 0.5 and rep["worst_group"] == "stand_shadow", rep)
check("the gap is flagged", "stand_shadow" in rep["flagged"] and "yellow_kit" in rep["flagged"], rep["flagged"])
check("a group that is fine is not flagged", "normal" not in rep["flagged"] and "muddy_ball" not in rep["flagged"])
n = sum(r["n"] for r in res.values()); f = sum(r["found"] for r in res.values())
card = {"name": "Ball finder (colour)", "version": "0.1", "date": "2026-10-19", "authors": "Team Hawks",
"purpose": "Find the ball in school football video so a goal trigger can work.",
"out_of_scope": "Not for refereeing. Not for judging individual players or children.",
"data": "Synthetic scenes drawn by code, 120 frames per situation.",
"overall": {"rate": rep["overall"], "found": f, "n": n, "tolerance_px": 6},
"slices": res,
"limitations": "Fails when part of the pitch is in shadow (stand shadow: 42%). Confused by a ball-coloured bib (yellow kit: 67%). Not tested on real footage.",
"people_and_privacy": "Works on balls and does not identify or track people. Videos of children stay on school machines.",
"contact": "hawks@example.school"}
check("a complete card has no problems", lint(card) == [], lint(card))
c2 = dict(card, limitations="")
check("empty limitations are caught", any("limitations" in p for p in lint(c2)), lint(c2))
c3 = dict(card, limitations="Works fine in most conditions.")
check("hiding the worst group is caught", any("worst group" in p for p in lint(c3)), lint(c3))
c4 = dict(card, purpose="TODO write this")
check("placeholder text is caught", any("placeholder" in p for p in lint(c4)), lint(c4))
c5 = dict(card, slices={"normal": res["normal"], "dim": res["normal"]})
check("fewer than 3 groups is caught", any("at least 3" in p for p in lint(c5)), lint(c5))
c6 = dict(card); del c6["people_and_privacy"]
check("missing people section is caught", any("people" in p for p in lint(c6)))
md = render(card)
check("rendered card has the table and sections", "| stand shadow |" in md and "## Known limitations" in md and "## People and privacy" in md)
open("MODEL_CARD.md", "w").write(md)
from data_audit import audit
files = [f"ball_s1_{i}.jpg" for i in range(60)] + [f"ball_s2_{i}.jpg" for i in range(5)] + [f"player_s1_{i}.jpg" for i in range(120)] + [f"goal_s1_{i}.jpg" for i in range(10)] + ["readme.txt"]
a = audit(files)
check("audit counts 195 images", a["total"] == 195, a["total"])
check("audit flags the rare class and session", any("goal" in n for n in a["notes"]) and any("s2" in n for n in a["notes"]), a["notes"])
check("audit flags a bad file name", any("do not follow" in n for n in a["notes"]))
check("audit flags a single-session dataset", any("ONE session" in n for n in audit([f"ball_s1_{i}.jpg" for i in range(10)])["notes"]))
check("audit of nothing does not crash", audit([])["total"] == 0)
print("failures:", bad)All sixteen should print PASS. Notice which cases test the card: an empty limitations section, a card that hides the worst group, placeholder text, fewer than three groups, and a missing people section are all caught.
Cell 5: the card this produced
# Model card: Ball finder (colour)
Version 0.1 · 2026-10-19 · Team Hawks
## What it is for
Find the ball in school football video so a goal trigger can work.
## What it is NOT for
Not for refereeing. Not for judging individual players or children.
## Data
Synthetic scenes drawn by code, 120 frames per situation.
## How well it works
Overall: **82%** (490 of 600 test frames, ball found within 6 px).
**By situation** (the overall number hides these):
| Situation | Found | Rate |
|---|---|---|
| normal | 120 of 120 | 100% |
| small ball | 120 of 120 | 100% |
| stand shadow | 50 of 120 | 42% |
| yellow kit | 80 of 120 | 67% |
| muddy ball | 120 of 120 | 100% |
## Known limitations
Fails when part of the pitch is in shadow (stand shadow: 42%). Confused by a ball-coloured bib (yellow kit: 67%). Not tested on real footage.
## People and privacy
Works on balls and does not identify or track people. Videos of children stay on school machines.
## Contact
hawks@example.schoolNow write your own for your team's model. Use your real numbers, including the bad ones. A card with only good numbers is an advertisement.
Questions: which group did we not test? Who might it fail for that we haven't thought of? Is it fair that the finder works on the sunny side of the pitch and not the shady side? Who decides what "fair enough" is?
If it goes wrong
| Error | Cause | Fix |
|---|---|---|
| Everything looks fine | Only tested one kind of situation | Make groups from what really varies: light, place, kit, camera |
| Groups have tiny sample sizes | Too few examples per group | Say so on the card; gather more before claiming anything |
| Lint says 'worst group not mentioned' | The limitation is vague | Name the group and its number |
| Team wants to label children by skin colour or gender | Misunderstanding of what fairness testing needs | Stop. Explain lesson 14 and section 4. Test situations, not children. |
| Card says 'works great' | Cherry-picked numbers | Add the per-group table and the worst number |
| Data audit shows one session only | All data from one day | Plan a second collection (lesson 28, lesson 39) |
| Card includes real names or faces | Over-sharing | Remove them. The card is public. |
| Your team disagrees about what is fair | Normal. There are several definitions. | Write down the definition your team chose and why |
Try it
- Add a fourth slice from your own footage
- Try to fix the worst group, then re-test and update the card
- Read a published model card (for any public model) and compare it with yours
Check what you learnt
What can an overall average hide?
Which of these is a source of bias?
Why does a model card have a 'what it is NOT for' section?
In this course, do we test the system on children's skin colour?
The worst-performing situation can't be fixed. What do you do?
Conclusion
No model is neutral: bias comes from data, labels, measurement, deployment and feedback. Averages hide groups, so measure results per situation, report the worst one, and fix what you can. The model card tells users what the model is for, how well it works where, and what it must not be used for.
What next?
Next: Pose estimation for sports analysis. You'll go from boxes to bodies with pose estimation.