Clean the Field Data & Retrain

Introduction

You're back from the field with hundreds of frames. Not all of them deserve a place in your dataset. Today you'll clean the catch (blurry frames, near-copies, broken labels), split it fairly, retrain your detector on real field data, and compare it with your classroom model on the same test set, then explain the difference honestly.

By the end of this lesson you'll be able to:

Clean a raw field dataset: blurry frames, near-duplicates, label errors
Explain why we move rejected files aside instead of deleting them
Check class balance after cleaning and decide what to do about it
Split field data by session (lesson 13) so the test set is honest
Retrain the detector on the field data and compare it fairly with the classroom model
Explain the gap between classroom and field performance with evidence

Working principle

The cleaning steps, in order

Step	Checks	Tool	Action
1. Back up	Is the raw data copied elsewhere?	You	Do this first
2. Unreadable and blurry	Variance of the Laplacian below a threshold (lesson 10)	`clean_dataset.py`	Move to `rejected/`
3. Near-duplicates	Almost identical images within one session	Perceptual hash	Keep the sharpest, move the rest
4. Label problems	Missing files, unknown classes, boxes outside the image, tiny boxes	`clean_dataset.py`	Move for relabelling
5. Class balance	Does one class have far fewer boxes?	Report	Collect or label more of it (lesson 20)
6. Split by session	Whole sessions in train, valid or test (lesson 13)	Split script	No session in two sets

Why move, not delete

A threshold that is a bit too strict can throw away good data. By moving rejects into a rejected/ folder with a reason in a report, you can check, undo, and explain what you did.

Near-duplicate detection with a perceptual hash

A dHash shrinks an image to 9 by 8 grey pixels and records, for each pixel, whether it is brighter than its right neighbour. That gives 64 yes/no values describing the image's overall shape. Two nearly identical frames give almost the same 64 values: the number of differing values is the Hamming distance. At 4 or less we call it a near-duplicate. We only compare frames from the same session, because frames from different positions are not duplicates even if they look alike.

A fair comparison

It is tempting to compare "mAP on the classroom validation set" with "mAP on the field validation set". But those are different exams. To compare two models, score both on the same held-out test set, ideally made of field sessions neither model has seen. Otherwise you may be measuring a harder exam, not a worse model.

Why field performance usually drops

Cause	Looks like
Distribution shift	Classroom images do not look like the pitch
Motion blur and distance	Ball is a few blurry pixels
Lighting	Sun, shadows, glare
Label noise	Rushed labels on hard frames
Small test set	Large swings from a few frames
Class imbalance	Too few balls

What you'll need

Your field dataset from lesson 28
Your cv_course environment with Ultralytics
Your lesson 24 model, to compare against
Google Colab for retraining

Build it

Step 1: clean the field data

Put the field images and YOLO labels in two flat folders: field_yolo/images/ and field_yolo/labels/. Roboflow exports have train, valid and test subfolders; do the cleaning before splitting, on the combined raw export, or run it once per folder. The file names must keep the capture session id (player_20261105-101500_0007...), which capture.py (lesson 11) puts there.

import csv
import os
import shutil
import sys
from collections import Counter
 
import cv2
import numpy as np
 
ROOT = sys.argv[1] if len(sys.argv) > 1 else "field_yolo"        # folder with images/ and labels/ (flat)
IMAGES, LABELS = os.path.join(ROOT, "images"), os.path.join(ROOT, "labels")
REJECTED = os.path.join(ROOT, "rejected")                          # we MOVE bad files here, never delete
NAMES = {0: "player", 1: "ball", 2: "goal"}                        # same order as data.yaml
BLUR_MIN = 40.0                                                    # variance of Laplacian (lesson 10)
DUP_DISTANCE = 4                                                   # dHash distance at or below this = near-duplicate
MIN_BOX = 0.004                                                    # smaller than this fraction of the image: suspicious
 
 
def dhash(gray, size=8):
    small = cv2.resize(gray, (size + 1, size))
    return (small[:, 1:] > small[:, :-1]).flatten()                # 64 true/false values describing the picture's gradients
 
 
def session_of(name):
    parts = name.split("_")
    return parts[1] if len(parts) > 2 else "unknown"               # player_20261105-101500_0007.jpg -> 20261105-101500
 
 
def label_problems(path):
    if not os.path.exists(path):
        return ["no label file"]
    problems, n = [], 0
    for line in open(path):
        p = line.split()
        if not p:
            continue
        n += 1
        if len(p) != 5:
            problems.append("bad line")
            continue
        cls, (xc, yc, w, h) = int(float(p[0])), map(float, p[1:])
        if cls not in NAMES:
            problems.append(f"unknown class {cls}")
        if not (0 <= xc <= 1 and 0 <= yc <= 1 and 0 < w <= 1 and 0 < h <= 1):
            problems.append("box outside image")
        elif w < MIN_BOX or h < MIN_BOX:
            problems.append("tiny box")
    return problems
 
 
def main():
    files = sorted(f for f in os.listdir(IMAGES) if f.lower().endswith((".jpg", ".jpeg", ".png")))
    info = {}
    for f in files:
        img = cv2.imread(os.path.join(IMAGES, f))
        if img is None:
            info[f] = {"blur": 0.0, "hash": None}
            continue
        gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
        info[f] = {"blur": float(cv2.Laplacian(gray, cv2.CV_64F).var()), "hash": dhash(gray)}
 
    reasons = {}                                                   # file -> why it is rejected
 
    for f in files:                                                # 1. unreadable or blurry
        if info[f]["hash"] is None:
            reasons[f] = "unreadable image"
        elif info[f]["blur"] < BLUR_MIN:
            reasons[f] = f"blurry ({info[f]['blur']:.0f})"
 
    by_session = {}                                                # 2. near-duplicates, compared only inside one session
    for f in files:
        by_session.setdefault(session_of(f), []).append(f)
    for session, group in by_session.items():
        kept = []
        for f in sorted(group, key=lambda x: -info[x]["blur"]):    # sharpest first, so the sharpest copy survives
            if f in reasons or info[f]["hash"] is None:
                continue
            twin = next((k for k in kept if np.count_nonzero(info[f]["hash"] != info[k]["hash"]) <= DUP_DISTANCE), None)
            if twin:
                reasons[f] = f"near-duplicate of {twin}"
            else:
                kept.append(f)
 
    for f in files:                                                # 3. label problems
        if f in reasons:
            continue
        problems = label_problems(os.path.join(LABELS, os.path.splitext(f)[0] + ".txt"))
        if problems:
            reasons[f] = "label: " + ", ".join(sorted(set(problems)))
 
    os.makedirs(REJECTED, exist_ok=True)
    with open(os.path.join(ROOT, "clean_report.csv"), "w", newline="") as out:
        w = csv.writer(out)
        w.writerow(["file", "action", "reason"])
        for f in files:
            if f in reasons:
                w.writerow([f, "moved to rejected/", reasons[f]])
                shutil.move(os.path.join(IMAGES, f), os.path.join(REJECTED, f))
                lab = os.path.join(LABELS, os.path.splitext(f)[0] + ".txt")
                if os.path.exists(lab):
                    shutil.move(lab, os.path.join(REJECTED, os.path.basename(lab)))
            else:
                w.writerow([f, "kept", ""])
 
    print(f"{len(files)} images checked: kept {len(files) - len(reasons)}, rejected {len(reasons)}")
    for kind, n in Counter(r.split(" (")[0].split(" of ")[0] for r in reasons.values()).most_common():
        print(f"  {kind}: {n}")
 
    counts = Counter()                                             # 4. class balance of what is left
    for f in os.listdir(LABELS):
        if f.endswith(".txt"):
            for line in open(os.path.join(LABELS, f)):
                if line.split():
                    counts[NAMES.get(int(float(line.split()[0])), "?")] += 1
    print("\nBoxes per class after cleaning:", dict(counts))
    if counts:
        lo, hi = min(counts.values()), max(counts.values())
        if hi > 5 * max(lo, 1):
            print(f"WARNING: imbalance {hi}:{lo}. The smallest class may be starved (lesson 20). Collect or label more of it.")
 
 
if __name__ == "__main__":
    main()

Run python clean_dataset.py field_yolo. It prints what it kept and rejected and why, writes field_yolo/clean_report.csv, and shows the number of boxes per class.

Questions: open five of the rejected files. Were all the decisions fair? If you change BLUR_MIN to 20 or DUP_DISTANCE to 8, what changes? Is the ball class a lot smaller than the player class? What will you do about it?

Step 2: fix labels and split

Upload the files listed as label problems back to Roboflow (or open them in your labelling tool) and correct them.
Re-export in YOLO format.
Split by session (the lesson 13 split_sessions function, or put whole sessions into the three Roboflow sets). Keep all frames of a session together.
Run the lesson 12 label checker on the final labels.

Step 3: retrain in Colab

Use the lesson 24 notebook with the new dataset. Two experiments make the comparison interesting. Give each a different name so the results do not overwrite each other:

from ultralytics import YOLO
 
# Experiment A: start from the pretrained COCO weights, train on the field data only
a = YOLO("yolov8n.pt")
a.train(data=f"{dataset.location}/data.yaml", epochs=50, patience=15, imgsz=640, project="football", name="field_only")
 
# Experiment B: start from your classroom model and keep training on the field data
b = YOLO("football/run1/weights/best.pt")           # the best.pt from lesson 24
b.train(data=f"{dataset.location}/data.yaml", epochs=50, patience=15, imgsz=640, project="football", name="classroom_plus_field")

Step 4: compare the runs

Download each run's results.csv and run this. Pass name=path pairs.

import sys
 
import matplotlib
import pandas as pd
 
matplotlib.use("Agg")
import matplotlib.pyplot as plt
 
# usage: python compare_runs.py classroom=runs/run1/results.csv field=runs/run2/results.csv
runs = dict(a.split("=", 1) for a in sys.argv[1:]) or {"classroom": "run1/results.csv", "field": "run2/results.csv"}
 
fig, axes = plt.subplots(1, 2, figsize=(10, 3.6))
print(f"{'run':<12}{'epochs':>7}{'best mAP50':>12}{'at epoch':>10}{'best mAP50-95':>15}{'final recall':>14}")
for name, path in runs.items():
    df = pd.read_csv(path)
    df.columns = [c.strip() for c in df.columns]
    i = df["metrics/mAP50(B)"].idxmax()
    print(f"{name:<12}{len(df):>7}{df.loc[i, 'metrics/mAP50(B)']:>12.3f}{int(df.loc[i, 'epoch']):>10}"
          f"{df['metrics/mAP50-95(B)'].max():>15.3f}{df['metrics/recall(B)'].iloc[-1]:>14.3f}")
    axes[0].plot(df["epoch"], df["metrics/mAP50(B)"], label=name)
    axes[1].plot(df["epoch"], df["val/box_loss"], label=name)
axes[0].set_title("mAP50 on each run's own validation set")
axes[1].set_title("validation box loss")
for a in axes:
    a.set_xlabel("epoch")
    a.legend()
plt.tight_layout()
plt.savefig("compare_runs.png", dpi=100)
print("\nCAUTION: these curves use each run's OWN validation set. To say that one model is better,")
print("score both models on the SAME held-out field test set (model.val(split='test') on one data.yaml).")

Then do the fair test: score both models on the same field test set, once:

from ultralytics import YOLO
 
test_yaml = f"{dataset.location}/data.yaml"      # its test: entry points at held-out field sessions
for name, weights in (("classroom model", "football/run1/weights/best.pt"),
                      ("field only", "football/field_only/weights/best.pt"),
                      ("classroom + field", "football/classroom_plus_field/weights/best.pt")):
    m = YOLO(weights).val(data=test_yaml, split="test")
    print(f"{name:<20} mAP50 {m.box.map50:.3f}   mAP50-95 {m.box.map:.3f}")

Questions: which model is best on the field test set? Is the classroom model worse on field data than on its own validation set? By how much? Which class (player, ball, goal) improved most?

Explain the gap: template

Each team writes one short paragraph for the dashboard of lesson 30 and the model card of lesson 43.

Line	Fill in
The gap	The classroom model scored ___ mAP50 on the classroom validation set but ___ on the field test set.
Evidence for the cause	I think the main reason is ___ because I saw ___ (for example: ball recall fell from ___ to ___; the ball is ___ pixels wide in field frames).
What retraining changed	After retraining, ___ improved by ___ and ___ did not change.
What we still can't claim	Because our test set has only ___ frames from ___ sessions, we cannot say ___.

If it goes wrong

Error	Cause	Fix
Almost everything is rejected as blurry	`BLUR_MIN` too high for outdoor images	Check the distribution of blur scores, then choose a threshold about at the 10th percentile
No near-duplicates found	Hash threshold too strict, or session ids missing	Raise `DUP_DISTANCE` to 6. Check file names contain the session.
Everything is in one session	File names lost the session id	Rebuild from the `capture_log.csv` (lesson 11), which has the session for each file
`no label file` for many images	Labels in a different folder or renamed	Check `LABELS` and the file stems
Class imbalance warning	Few balls labelled	Label more ball frames or capture more (lesson 39)
Retrained model is worse on the classroom set	It adapted to the field	Expected. Evaluate on a set that represents the real use.
Metrics are identical for two runs	Same weights evaluated twice	Check the `weights` paths
Colab ran out of time	Free GPU limits	Fewer epochs, or train in turns

Try it

Plot the distribution of blur scores per session and pick the threshold from it
Add a check for boxes that are far too big or too small for their class
Compute per-class precision and recall from the validation output

Check what you learnt

Why compare duplicates only within one session?
Why move rejected files instead of deleting them?
Why must two models be scored on the same test set?
Which is a reason field data can lower scores?
Your test set has only 40 frames. What does that mean?

Conclusion

Raw field data needs sorting before it's useful: remove blur and near-duplicates within each session, fix labels, and split by session. Retrain, score both models on the same test set, and explain any gap honestly: distribution shift, blur, distance and label noise are usually the reasons.

What next?

Next: Live match analytics dashboard: JS front end. You'll build a live match dashboard in JavaScript.