CNNs: How Convolutional Layers See

Introduction

The network from lesson 16 treats every pixel as a separate input, which falls apart on a real photo: too many weights, and it can't tell that a ball in the corner is the same as a ball in the middle. Convolutional neural networks (CNNs) fix both problems with one idea: slide a small filter over the picture. Today you'll do it by hand, in NumPy, and then peek inside a real network.

By the end of this lesson you'll be able to:

• Explain convolution as sliding a small filter over an image
• Say what a feature map is and why one filter finds one kind of pattern
• Explain ReLU and max-pooling and what each one throws away
• Describe why stacking layers lets a network go from edges to shapes to objects
• Implement convolution, ReLU and pooling in NumPy and check them against OpenCV
• Explain why a convolutional layer needs far fewer weights than a flat layer
• Show the feature maps of a real network for a sports photo

Working principle

Why not just use the flat network from lesson 16?

Flattening a picture into a list of numbers throws away where things are. A ball in the top-left and a ball in the bottom-right look completely different to a flat layer, and it would need separate weights for each. A 120 by 160 picture fed to just one neuron needs 19,200 weights.

A convolutional layer uses the same small filter everywhere: 9 numbers for a 3 by 3 filter. It learns "what an edge looks like" once and finds it anywhere in the picture.

Convolution

Place the 3 by 3 filter over a 3 by 3 patch of the image. Multiply each filter number by the pixel beneath it, add up the nine products. That sum is one pixel of the output. Slide by one pixel and repeat.

You met these in lesson 7 as hand-made filters. The big idea of a CNN: the network learns the filter numbers itself during training.

ReLU: keep what fired

After convolution, ReLU sets every negative value to 0 and keeps the positive ones. It's the network saying "only tell me where the pattern was found".

Max-pooling: shrink, keep the strongest

Pooling takes each 2 by 2 block and keeps only the largest value. The picture becomes half the width and half the height. We lose exact position but keep "there was an edge around here", which makes the network less fussy about small shifts, and cheaper.

Stacking layers

The first layers of nearly every vision network learn similar, simple things. That's why transfer learning (lesson 21) works.

A typical CNN recipe

image -> [convolution -> ReLU -> pool] x a few -> flatten -> dense layers -> answer

What you'll need

• Your cv_course environment with Jupyter
• TensorFlow for the last step: pip install tensorflow
• A football photo

Build it

Step 1: convolution, ReLU and pooling from scratch

This draws a small pretend football scene. To use a real photo instead, replace make_scene() with cv2.imread("football.jpg") and resize it to about 160 by 120.

import cv2
import matplotlib
import numpy as np
 
matplotlib.use("Agg")                      # remove this line to see plots in a notebook
import matplotlib.pyplot as plt
 
 
def make_scene():
    """A small pretend football scene: grass, a white line, a ball and a player."""
    img = np.zeros((120, 160, 3), np.uint8)
    img[:] = (40, 130, 40)
    cv2.line(img, (0, 80), (160, 80), (255, 255, 255), 2)
    cv2.circle(img, (110, 95), 7, (255, 255, 255), -1)
    cv2.rectangle(img, (30, 30), (44, 70), (200, 60, 30), -1)
    cv2.circle(img, (37, 24), 6, (150, 190, 230), -1)
    return img
 
 
scene = make_scene()
gray = cv2.cvtColor(scene, cv2.COLOR_BGR2GRAY).astype(np.float32) / 255
 
 
def convolve(image, kernel):
    """Slide the kernel over the image. Each output pixel = sum of (patch * kernel)."""
    kh, kw = kernel.shape
    h, w = image.shape
    out = np.zeros((h - kh + 1, w - kw + 1), np.float32)
    for y in range(out.shape[0]):
        for x in range(out.shape[1]):
            out[y, x] = np.sum(image[y:y + kh, x:x + kw] * kernel)
    return out
 
 
filters = {
    "vertical edges": np.array([[-1, 0, 1], [-2, 0, 2], [-1, 0, 1]], np.float32),
    "horizontal edges": np.array([[-1, -2, -1], [0, 0, 0], [1, 2, 1]], np.float32),
    "blur": np.ones((3, 3), np.float32) / 9,
    "sharpen": np.array([[0, -1, 0], [-1, 5, -1], [0, -1, 0]], np.float32),
}
 
 
def relu(x):
    return np.maximum(0, x)
 
 
def max_pool(x, size=2):
    h, w = x.shape[0] // size * size, x.shape[1] // size * size
    return x[:h, :w].reshape(h // size, size, w // size, size).max(axis=(1, 3))
 
 
fig, axes = plt.subplots(len(filters), 3, figsize=(7, 8))
for row, (name, k) in zip(axes, filters.items()):
    fmap = convolve(gray, k)
    act = relu(fmap)
    pooled = max_pool(act)
    row[0].imshow(fmap, cmap="gray")
    row[0].set_title(f"{name}: feature map", fontsize=7)
    row[1].imshow(act, cmap="gray")
    row[1].set_title("after ReLU", fontsize=7)
    row[2].imshow(pooled, cmap="gray")
    row[2].set_title(f"after pooling {pooled.shape}", fontsize=7)
    for a in row:
        a.axis("off")
    print(f"{name:18s} feature map {fmap.shape} -> pooled {pooled.shape}   max response {act.max():.2f}")
plt.tight_layout()
plt.savefig("featmaps.png", dpi=100)
 
# Does our slow loop agree with OpenCV's fast version?
k = filters["vertical edges"]
fast = cv2.filter2D(gray, -1, k, borderType=cv2.BORDER_CONSTANT)[1:-1, 1:-1]
print("matches OpenCV:", np.allclose(convolve(gray, k), fast, atol=1e-4))
 
# Why convolution is thrifty
print("A 3x3 filter has", k.size, "numbers. A dense layer from 120*160 pixels to 1 neuron has", 120 * 160, "weights.")

Expected output: each feature map is 118 by 158 (the image shrinks by 2 because the 3 by 3 window doesn't fit at the border), pooling makes it 59 by 79, and matches OpenCV: True.

Questions: which filter lights up the ball? The white line? The player's sides? What does the blur filter do to a feature map? Why is the feature map smaller than the image?

Step 2: invent your own filter

Add a filter to the filters dictionary and look at the result. Ideas: a diagonal edge [[0, 1, 2], [-1, 0, 1], [-2, -1, 0]], or a blob detector with a bright centre and dark ring. Predict what it will find before you run it.

Step 3: see a real network's feature maps (TensorFlow)

This needs pip install tensorflow and an internet connection the first time, to download the pretrained MobileNetV2 weights. Use any football photo. It shows what the early and middle layers pay attention to.

import cv2
import matplotlib
import numpy as np
import tensorflow as tf
 
matplotlib.use("Agg")
import matplotlib.pyplot as plt
 
PHOTO = "football.jpg"            # put a photo next to this script
LAYERS = ["block_1_expand_relu", "block_3_expand_relu", "block_6_expand_relu"]   # early, middle, deeper
 
model = tf.keras.applications.MobileNetV2(weights="imagenet", include_top=False, input_shape=(224, 224, 3))
probe = tf.keras.Model(inputs=model.input, outputs=[model.get_layer(n).output for n in LAYERS])
 
img = cv2.cvtColor(cv2.imread(PHOTO), cv2.COLOR_BGR2RGB)
img = cv2.resize(img, (224, 224))
x = tf.keras.applications.mobilenet_v2.preprocess_input(img.astype("float32"))[None]
 
outputs = probe.predict(x, verbose=0)
 
fig, axes = plt.subplots(len(LAYERS), 6, figsize=(12, 6))
for row, name, fmaps in zip(axes, LAYERS, outputs):
    print(name, "feature maps:", fmaps.shape)             # (1, height, width, channels)
    for i, ax in enumerate(row):
        ax.imshow(fmaps[0, :, :, i], cmap="viridis")
        ax.axis("off")
        if i == 0:
            ax.set_title(name, fontsize=7, loc="left")
plt.tight_layout()
plt.savefig("real_featmaps.png", dpi=100)

Questions: what do the early layers (top row) respond to? And the deeper layers? Do you see any map that looks like it follows the ball or the players? Why do the later maps look blockier?

If the layer names are rejected on your version of Keras, run model.summary() and pick three layers with expand_relu in their names.

If it goes wrong

Try it

• Implement a stride of 2 in convolve and describe the effect on the output size
• Stack two convolutions (edges, then edges of edges) and show the result
• Count the weights in a small CNN and compare with a dense network

Conclusion

A CNN slides small filters over the picture; each one makes a feature map showing where its pattern appears. ReLU keeps the strong answers, max-pooling shrinks the map, and stacking layers builds up from edges to shapes to objects, with far fewer weights than a flat network.

What next?

Next: Train your first classifier. You'll train your first real image classifier: ball or no ball.