Transfer Learning: Standing on Giants' Shoulders

Introduction

Big companies train networks on millions of photos for weeks. You don't have that, and you don't need it. With transfer learning you borrow one of those networks, keep everything it already knows about edges, shapes and textures, and teach it only the last step: your ball on your pitch. It needs fewer pictures and less time, and usually wins.

By the end of this lesson you'll be able to:

• Explain transfer learning: reusing what a network learned on a big dataset for a new, smaller job
• Say what freezing layers means, and why we do it
• Build a model with a frozen MobileNetV2 base and a small new head
• Train the head, then fine-tune the top of the base with a very small learning rate
• Compare it with the lesson 18 model on accuracy, training time and the amount of data needed
• Name the trade-offs: size, speed and fit to your task

Working principle

Why it works

From lesson 17: the first layers of a CNN learn simple things (edges, corners, colour blobs) that are useful for any picture. Only the last layers are specific to the original task (dogs, cars, cups).

So we keep the early and middle layers as they are, and replace the last layers with our own.

The recipe

The tiny learning rate matters: large steps would wreck what the base already knows.

What transfer learning buys you

Choosing a base model

All are in keras.applications. Check the Keras documentation for the exact names and input sizes.

Limits

• The base was trained on everyday photos. A very different domain (X-rays, thermal images) transfers less.
• Input size matters: with ImageNet weights MobileNetV2 expects 96, 128, 160, 192 or 224 pixels. Our 64 by 64 crops are enlarged to 96.
• Bias comes along too. The base inherits whatever was in its data (lesson 43).

What you'll need

• Your cv_course environment with TensorFlow
• Your crops from lesson 18
• An internet connection the first time (to download MobileNetV2's weights)

Build it

Install: pip install tensorflow matplotlib. Uses the crops/train and crops/valid folders from lesson 18 (real or synthetic). Do not open crops/test today.

import matplotlib
import numpy as np
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
 
matplotlib.use("Agg")
import matplotlib.pyplot as plt
 
IMG_SIZE = (96, 96)              # MobileNetV2 with ImageNet weights accepts 96, 128, 160, 192 or 224
BATCH = 32
CLASS_NAMES = ["no_ball", "ball"]
 
 
def load(split, shuffle):
    return keras.utils.image_dataset_from_directory(
        f"crops/{split}", class_names=CLASS_NAMES, label_mode="binary",
        image_size=IMG_SIZE, batch_size=BATCH, shuffle=shuffle, seed=42,
    )
 
 
train_ds = load("train", True).prefetch(tf.data.AUTOTUNE)
val_ds = load("valid", False).prefetch(tf.data.AUTOTUNE)
 
# 1. The borrowed brain: MobileNetV2 already trained on 1.4 million everyday photos (downloads once, ~10 MB)
base = keras.applications.MobileNetV2(input_shape=IMG_SIZE + (3,), include_top=False, weights="imagenet")
base.trainable = False                                  # FREEZE it: keep what it already knows
 
# 2. Our own small head, and a bit of augmentation (only active during training)
inputs = keras.Input(shape=IMG_SIZE + (3,))
x = layers.RandomFlip("horizontal")(inputs)
x = layers.RandomBrightness(0.2, value_range=(0, 255))(x)
x = layers.Rescaling(1 / 127.5, offset=-1)(x)           # MobileNetV2 expects pixels in -1..1
x = base(x, training=False)                             # keep the frozen network's batch-norm in inference mode
x = layers.GlobalAveragePooling2D()(x)
x = layers.Dropout(0.3)(x)
outputs = layers.Dense(1, activation="sigmoid")(x)
model = keras.Model(inputs, outputs)
 
trainable = sum(int(np.prod(w.shape)) for w in model.trainable_weights)
total = model.count_params()
print(f"parameters: {total:,} in total, but only {trainable:,} are being trained")
 
# 3. Train just the head (fast)
model.compile(optimizer=keras.optimizers.Adam(1e-3), loss="binary_crossentropy", metrics=["accuracy"])
h1 = model.fit(train_ds, validation_data=val_ds, epochs=5)
 
# 4. Fine-tune: unfreeze the top of the borrowed network, with a LOW learning rate
base.trainable = True
for layer in base.layers[:-30]:
    layer.trainable = False                             # keep the early edge-finders fixed
model.compile(optimizer=keras.optimizers.Adam(1e-5), loss="binary_crossentropy", metrics=["accuracy"])
h2 = model.fit(train_ds, validation_data=val_ds, epochs=5)
 
model.save("ball_mobilenet.keras")
 
acc = h1.history["val_accuracy"] + h2.history["val_accuracy"]
plt.plot(acc, marker="o")
plt.axvline(len(h1.history["val_accuracy"]) - 0.5, color="grey", linestyle="--")
plt.xlabel("epoch (dashed line: fine-tuning starts)")
plt.ylabel("validation accuracy")
plt.savefig("transfer_curve.png", dpi=100)
print("best validation accuracy:", max(acc))

You should see something like: the head-only model reaches a high validation accuracy within 1 or 2 epochs, much faster than the lesson 18 model, and fine-tuning adds a little more, or sometimes nothing. Your numbers will depend on your data.

Comparison table

Fill this in with your own numbers.

Questions: did the pretrained model need fewer epochs? Fewer images? Try using only 10% of the training crops with both models: which one suffers more? Why do we use a tiny learning rate for fine-tuning?

If it goes wrong

Try it

• Swap in EfficientNetB0 and compare size, speed and accuracy
• Unfreeze 10, 30 and 60 layers and plot the validation accuracy for each
• Visualise which crops the model gets wrong and why (lesson 19 error analysis)

Conclusion

Transfer learning reuses a network trained on a huge dataset: freeze the base, train a small new head for your classes, then fine-tune the top layers with a tiny learning rate. You get better accuracy from fewer pictures and less training, at the cost of a bigger model that wasn't built for your exact task.

What next?

Next: Model to the browser: TensorFlow.js. You'll put a model inside a web page, running on the visitor's own device.