What Makes Good Sports Data?

Introduction

From Unit 4 on you'll train AI models. A model learns only from the examples you give it, so the most important work happens before training. Today you'll learn what makes a good football dataset, measure blur and brightness with code, catch sneaky duplicate frames, and audit a dataset like a professional.

By the end of this lesson you'll be able to:

• Explain why a model is only as good as its data
• Name the main data quality problems in field sports: blur, occlusion, lighting, scale, angle and class imbalance
• Measure blur and brightness with code
• Find near-duplicate frames and explain why they cause a false sense of success
• Audit a dataset and write up 5 specific problems with evidence
• Describe what "diverse" means for a football dataset

Working principle

Garbage in, garbage out

A model learns patterns from examples. If the examples are narrow, blurry, wrongly labelled or repeated, the model learns the wrong thing. You can't fix poor data with a clever model.

Quality problems in football data

Diversity

A good dataset covers the situations the model will meet: different angles, distances, lighting, teams and kits, pitch surfaces, weather, players (age, height, skin tone), and moments (play, set pieces, celebrations, empty pitch). If a group is missing, the model will be worse on it. This comes back in lesson 43.

The duplicate trap

Video gives you 30 almost identical frames per second. If you shuffle all the frames and split them into training and test sets, near-identical frames land in both. The model "remembers" the test frames, and the score looks perfect. This is called leakage. In lesson 13 we fix it by splitting by video, not by frame.

Measuring quality in code

• Blur score: the variance of the Laplacian. A sharp image has strong edges, so a high variance. A blurry one is low. As a rough guide, below about 80 is blurry, but check on your own images.
• Brightness: the mean of the grayscale pixels. Below about 50 is very dark. Above about 205 is washed out.
• Near-duplicates: shrink each image to 8 by 8 grayscale and compare tiny fingerprints (an "average hash"). Few differing bits means the same scene.

What you'll need

• Your cv_course environment
• A folder of football frames to audit (audit.py makes a small pretend dataset with planted problems if you don't have one)

Build it

It scans a folder of images and writes a table with blur, brightness, size and a "same as an earlier image" flag. Without a folder, it makes a small fake dataset so the lab runs.

import csv
import os
import sys
 
import cv2
import numpy as np
 
BLUR_MIN = 80.0          # below this: blurry
DARK_MAX = 50.0          # mean brightness below this: too dark
BRIGHT_MIN = 205.0       # above this: washed out
MIN_SIDE = 320           # smaller than this on either side: too small
HASH_BITS_SAME = 4       # fingerprints differing by this many bits or fewer: near-duplicate
 
 
def average_hash(gray):
    small = cv2.resize(gray, (8, 8), interpolation=cv2.INTER_AREA)
    return (small > small.mean()).flatten()
 
 
def make_fake_dataset(folder):
    os.makedirs(folder, exist_ok=True)
    rng = np.random.default_rng(0)
    for i in range(8):
        img = np.zeros((360, 640, 3), np.uint8)
        img[:] = (34, 142, 28)
        for k in range(60):                                   # lots of edges so it is "sharp"
            x, y = int(rng.integers(0, 600)), int(rng.integers(0, 320))
            cv2.rectangle(img, (x, y), (x + 30, y + 30), (int(rng.integers(0, 255)),) * 3, -1)
        if i == 3:
            img = cv2.GaussianBlur(img, (31, 31), 0)          # a blurry one
        if i == 5:
            img = (img * 0.35).astype(np.uint8)               # a dark one
        cv2.imwrite(os.path.join(folder, f"img_{i:02d}.jpg"), img)
        if i == 1:                                            # a near-duplicate of image 1
            cv2.imwrite(os.path.join(folder, "img_01_copy.jpg"), img)
 
 
def audit(folder, out_csv="audit.csv"):
    files = sorted(f for f in os.listdir(folder) if f.lower().endswith((".jpg", ".jpeg", ".png")))
    rows = []
    hashes = []
    for name in files:
        img = cv2.imread(os.path.join(folder, name))
        if img is None:
            rows.append([name, "", "", "", "", "unreadable", ""])
            continue
        h, w = img.shape[:2]
        gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
        blur = cv2.Laplacian(gray, cv2.CV_64F).var()
        bright = float(gray.mean())
        hsh = average_hash(gray)
 
        flags = []
        if blur < BLUR_MIN:
            flags.append("blurry")
        if bright < DARK_MAX:
            flags.append("dark")
        if bright > BRIGHT_MIN:
            flags.append("washed-out")
        if min(h, w) < MIN_SIDE:
            flags.append("small")
 
        dup_of = ""
        for other_name, other_hash in hashes:
            if int(np.count_nonzero(hsh != other_hash)) <= HASH_BITS_SAME:
                dup_of = other_name
                flags.append("duplicate")
                break
        hashes.append((name, hsh))
 
        rows.append([name, f"{w}x{h}", round(blur, 1), round(bright, 1), dup_of, ";".join(flags) or "ok"])
 
    with open(out_csv, "w", newline="") as f:
        wr = csv.writer(f)
        wr.writerow(["file", "size", "blur_score", "brightness", "duplicate_of", "flags"])
        wr.writerows([r[:6] for r in rows])
 
    bad = [r for r in rows if r[5] != "ok"]
    print(f"{len(rows)} images, {len(bad)} with problems")
    for r in bad:
        print(" ", r[0], "->", r[5])
    return rows
 
 
if __name__ == "__main__":
    folder = sys.argv[1] if len(sys.argv) > 1 else "fake_dataset"
    if not os.path.isdir(folder):
        make_fake_dataset(folder)
    audit(folder)

Run it as python audit.py (fake data) or python audit.py path/to/images.

Your audit sheet

Also count the classes in the dataset's labels (player, ball, goal) and note which one is rarest. That is issue number one in most football sets.

If it goes wrong

Try it

• Plot a histogram of the blur scores and pick a threshold
• Count how many labelled balls and players the dataset has, and compute the imbalance
• Add a check for tiny images, or a check for too-narrow angle variety

Conclusion

A model is only as good as its data. Watch for blur, occlusion, lighting, scale, angle, class imbalance and near-duplicates, and measure them in code instead of guessing. A diverse dataset (different light, angles, distances and players) is what makes a model work in a real, messy match.

What next?

Next: Webcam data capture pipeline. You'll write a capture script that collects good data in the first place.