Train, Validation & Test Splits, and Augmentation

Introduction

You've got labelled frames. Before training, they need splitting into three piles: one to learn from, one to check progress, and one to keep secret for the final test. Split them wrongly and your model will look brilliant in class and fail on match day. Today you'll split the right way, then use augmentation to get more variety out of the same photos.

By the end of this lesson you'll be able to:

• Explain what the training, validation and test sets are for
• Split a dataset 70/20/10 by session, so near-identical frames don't leak
• Keep class proportions similar in each split
• Choose augmentations that make sense for football (motion blur, brightness, crop, flip) and reject those that don't
• Apply augmentation only to the training set, with the boxes updated correctly
• Export the final dataset in YOLO format

Working principle

Three sets, three jobs

Never tune on the test set. If you do, it becomes part of training, and the score lies.

Leakage: split by session, not by frame

From lesson 10: frames from the same video are almost identical. A random split puts a frame in train and its twin in test, and the score is fake.

Fix: keep every frame of one session (one camera position and time, the session id in the file name from lesson 11) together in one split. Then the test set is genuinely new places and moments.

This needs enough sessions. With only 3 sessions you can't make a 70/20/10 split. Aim for at least 10 per class. That is one more reason the field trip uses several camera positions.

Keeping classes balanced

Split each class separately, so every split has players, balls and goals in similar proportions. (This is called a stratified split.)

Augmentation: more variety from the same data

Rule: only augment the training set. Validation and test must stay as real as possible.

Augmentation must move the boxes too

If you crop or flip an image, the labels must change with it. Libraries such as Albumentations do this when you give them the boxes.

What you'll need

• Your labelled dataset from lesson 12 (or the prepared practice dataset)
• Your cv_course environment
• Your Roboflow project, for the no-code way

Build it

Step 1: split by session

import random
from collections import defaultdict
 
 
def session_of(filename):
    """player_20261105-101500_0007.jpg -> 20261105-101500"""
    return filename.split("_")[1]
 
 
def split_sessions(files, ratios=(0.7, 0.2, 0.1), seed=42):
    """Split one class's files so every session stays whole in one set."""
    by_session = defaultdict(list)
    for f in files:
        by_session[session_of(f)].append(f)
 
    sessions = list(by_session.items())
    random.Random(seed).shuffle(sessions)
 
    names = ["train", "val", "test"]
    total = len(files)
    result = {n: [] for n in names}
    for _, items in sessions:
        # give the session to the set that is furthest below its target share
        deficits = [ratios[i] - len(result[n]) / total for i, n in enumerate(names)]
        target = names[deficits.index(max(deficits))]
        result[target].extend(items)
    return result
 
 
# Try it on pretend files: 12 sessions of 10 frames each
files = [f"player_20261105-{h:02d}0000_{i:04d}.jpg" for h in range(10, 22) for i in range(1, 11)]
parts = split_sessions(files)
 
for name, items in parts.items():
    print(name, len(items), "frames from", len({session_of(f) for f in items}), "sessions")
 
# The leakage check: no session may appear in two sets
train_s = {session_of(f) for f in parts["train"]}
val_s = {session_of(f) for f in parts["val"]}
test_s = {session_of(f) for f in parts["test"]}
print("overlap train/val:", train_s & val_s)
print("overlap train/test:", train_s & test_s)
print("overlap val/test:", val_s & test_s)

Question: the shares aren't exactly 70, 20 and 10. Why? What would you need to make them closer?

Apply it to a real dataset (one class at a time, so each class is split separately):

import os
import shutil
 
ROOT = "dataset"                 # from lesson 11: dataset/<class>/<files>
OUT = "dataset_split"
 
for cls in sorted(os.listdir(ROOT)):
    folder = os.path.join(ROOT, cls)
    if not os.path.isdir(folder):
        continue
    files = sorted(f for f in os.listdir(folder) if f.endswith(".jpg"))
    parts = split_sessions(files)
    for split, items in parts.items():
        os.makedirs(os.path.join(OUT, split, cls), exist_ok=True)
        for f in items:
            shutil.copy(os.path.join(folder, f), os.path.join(OUT, split, cls, f))
    print(cls, {k: len(v) for k, v in parts.items()})

Step 2: augment the training images (with boxes)

Install it first: pip install albumentations. Albumentations renames or changes some arguments between versions (for example MotionBlur, Rotate and RandomSizedBBoxSafeCrop). If you get an error about an unknown argument, check the transform's page in the Albumentations documentation for your installed version (pip show albumentations).

import albumentations as A
import cv2
import numpy as np
 
# One picture with one ball and one player (boxes in YOLO format)
img = np.zeros((360, 640, 3), np.uint8)
img[:] = (34, 142, 28)
cv2.circle(img, (320, 180), 9, (0, 140, 255), -1)
cv2.circle(img, (200, 220), 16, (200, 60, 30), -1)
boxes = [(320 / 640, 180 / 360, 20 / 640, 20 / 360),          # ball
         (200 / 640, 220 / 360, 34 / 640, 34 / 360)]          # player
labels = ["ball", "player"]
 
augment = A.Compose(
    [
        A.RandomBrightnessContrast(brightness_limit=0.3, contrast_limit=0.2, p=0.7),
        A.MotionBlur(blur_limit=(3, 7), p=0.3),
        A.HorizontalFlip(p=0.5),
        A.Rotate(limit=5, p=0.3),
        A.RandomSizedBBoxSafeCrop(height=360, width=640, p=0.4),   # crop but never cut a box off
    ],
    bbox_params=A.BboxParams(format="yolo", label_fields=["labels"], min_visibility=0.3),
)
 
for i in range(4):
    out = augment(image=img, bboxes=boxes, labels=labels)
    pic = out["image"].copy()
    h, w = pic.shape[:2]
    for (xc, yc, bw, bh), name in zip(out["bboxes"], out["labels"]):
        x1, y1 = int((xc - bw / 2) * w), int((yc - bh / 2) * h)
        x2, y2 = int((xc + bw / 2) * w), int((yc + bh / 2) * h)
        cv2.rectangle(pic, (x1, y1), (x2, y2), (0, 255, 0), 1)
        cv2.putText(pic, name, (x1, y1 - 3), cv2.FONT_HERSHEY_SIMPLEX, 0.4, (0, 255, 0), 1, cv2.LINE_AA)
    cv2.imwrite(f"aug_{i}.jpg", pic)
    print(i, "boxes after augmentation:", len(out["bboxes"]))

Question: do the boxes still sit on the ball and the player in every output? What happens to bboxes when a crop removes the ball?

Step 3: the no-code way in Roboflow

1. Open your project, then Generate, and set the split to 70 / 20 / 10 (Roboflow offers train, valid, test). Note that Roboflow splits by image, so for field data split by session yourself first, or upload the three groups into their own sets.
2. Add augmentation only to the training set: brightness, blur, flip and crop. Roboflow applies augmentation to training images only.
3. Choose the YOLOv8 format and Download the zip. Check data.yaml for the class names and the paths for train, valid and test.

If it goes wrong

Try it

• Compare a random frame split with a session split: count near-duplicates across train and test using audit.py from lesson 10
• Write a custom augmentation (simulate fog with a white overlay)
• Show a grid of 12 augmented images and rate how realistic each is

Conclusion

Three sets, three jobs: train to learn, validation to tune, test once at the end. Split by session so near-identical frames can't leak, keep class proportions similar, and augment the training set only, with realistic changes that move the boxes too.

What next?

Next: Consent, privacy & filming people. You'll look at the people in your data: consent and privacy.