Computer Vision to Arduino or ESP32

Introduction

Until now your detector's reactions stayed on the screen. Today it reaches into the real world: a goal flashes an LED strip, sounds a buzzer, or moves a servo flag, through an Arduino or ESP32 plugged into your laptop. You'll design the conversation between Python and the board, and make sure the match goes on even if someone trips over the cable.

By the end of this lesson you'll be able to:

Describe a serial protocol as a table of commands and replies
Write a sketch that listens without blocking
Send commands from Python and read the replies
Handle a missing, silent or unplugged board without crashing
Test the Python side with a pretend board
State two safety rules for hardware in the classroom

Working principle

Software meets the world

Until now an event (a goal) changed a number on a screen. Today it changes the world: a light and a sound in the room. The idea is the same as lesson 31; we add one more bus.on("goal", ...) handler, and this one talks to hardware.

Serial: a conversation, one line at a time

The USB cable carries bytes in both directions at an agreed speed (the baud rate, 115200). We agree a tiny language of lines:

Python sends	Board answers	Meaning
`PING`	`PONG`	Are you there?
`GOAL`	`OK GOAL`	Alarm for 1.5 s
`SCORE 3`	`OK SCORE 3`	Blink three times
`RESET`	`OK RESET`	Everything off
anything else	`ERR unknown`	I did not understand

This is a protocol. Both sides must follow it exactly. Writing it as a table first, before any code, saves hours.

Always acknowledge

Every command gets a reply. Without it, Python cannot tell "worked" from "board not listening". With it, Python can retry, warn, or give up gracefully.

The sketch must keep listening

A common mistake is delay(1500) to hold the alarm. While the board is delayed it hears nothing. Instead the sketch notes when the alarm should end (millis()) and checks the clock on each pass of loop(). This is the same idea as the doorbell in lesson 31: don't busy-wait, react.

The one delay() we kept (the score blinks) is short. So the sketch replies first and blinks after, so Python's half-second wait is never exceeded.

Hardware fails, and the match must go on

Cables are unplugged, boards reset, a port belongs to another program. The Python side treats the board as optional:

Failure	What happens
No pyserial / no port	Runs without hardware, says so
Silent board	Retries once, then reports
Cable pulled mid-match	Catches the error, switches to no-board mode
Board says `ERR`	Logs the command that was not understood

If a goal log fails because a buzzer is unplugged, that is a bad design. The log (lesson 31) and the board are separate handlers for exactly this reason.

Arduinos reset when the port opens

Many boards (Uno, Nano) restart when a program opens the port. If you send a command at once, it is lost. We wait two seconds after opening, then clear the input buffer.

Uno versus ESP32

The same sketch runs on both with different pin numbers. The ESP32 runs at 3.3 V, which is fine for an LED with a resistor. An active buzzer rated for 5 V may be quiet on 3.3 V. ESP32 can also do Wi-Fi, so later you could send the same commands over the network (an extension) instead of a cable.

Safety

Keep to low-voltage parts on a breadboard. Never connect mains-powered devices (a big floodlight, a siren) directly. If students want to switch a bigger load, that needs a properly rated relay module and a teacher's check. Buzzers can also be loud: keep them away from ears and warn the room.

What you'll need

An Arduino Uno or ESP32 with a USB cable, and the Arduino IDE
An LED (with a 220 Ω resistor), a buzzer, or a servo
Your cv_course environment with pyserial: pip install pyserial
Your goal detector from lesson 31

Build it

You need: an Arduino Uno/Nano or an ESP32 board, a USB cable, one LED with a 220 ohm resistor on pin 8, and an active buzzer on pin 9 (the on-board LED is used for the score). Install pyserial: pip install pyserial. No board? Everything except Cell 5 runs with the pretend board in Cell 4.

Wiring

Pin 8 ----[220 ohm]----(LED, long leg)----(LED, short leg)---- GND
Pin 9 ---- buzzer (+) ,  buzzer (-) ---- GND          (active buzzer)

Cell 1: the sketch (upload it from the Arduino IDE)

// lesson 35: the goal board. Python sends one line per command, this sketch answers one line.
// Works on an Arduino Uno/Nano and on an ESP32 (change the three pin numbers).
//
//   PING       -> PONG
//   GOAL       -> OK GOAL     (LED + buzzer for about 1.5 seconds)
//   SCORE 3    -> OK SCORE 3  (the LED blinks 3 times, to show the score)
//   RESET      -> OK RESET
//   anything else -> ERR unknown
 
const int LED_PIN = 8;        // an LED with a 220 ohm resistor to GND
const int BUZZER_PIN = 9;     // an ACTIVE buzzer (it beeps by itself when the pin is HIGH)
const int STATUS_PIN = 13;    // the on-board LED. On many ESP32 boards use 2.
 
String line = "";
unsigned long alarmUntil = 0;   // when to switch the alarm off (millis)
 
void setup() {
  pinMode(LED_PIN, OUTPUT);
  pinMode(BUZZER_PIN, OUTPUT);
  pinMode(STATUS_PIN, OUTPUT);
  Serial.begin(115200);
}
 
void startAlarm(unsigned long ms) {
  digitalWrite(LED_PIN, HIGH);
  digitalWrite(BUZZER_PIN, HIGH);
  alarmUntil = millis() + ms;
}
 
void blinkScore(int n) {
  for (int i = 0; i < n && i < 10; i++) {
    digitalWrite(STATUS_PIN, HIGH);
    delay(150);
    digitalWrite(STATUS_PIN, LOW);
    delay(150);
  }
}
 
void handle(String cmd) {
  cmd.trim();
  if (cmd == "PING") {
    Serial.println("PONG");
  } else if (cmd == "GOAL") {
    startAlarm(1500);
    Serial.println("OK GOAL");
  } else if (cmd.startsWith("SCORE ")) {
    int n = cmd.substring(6).toInt();
    Serial.print("OK SCORE ");        // answer FIRST: Python only waits half a second for a reply
    Serial.println(n);
    blinkScore(n);                    // then blink (this blocks for 0.3 s per blink)
  } else if (cmd == "RESET") {
    digitalWrite(LED_PIN, LOW);
    digitalWrite(BUZZER_PIN, LOW);
    alarmUntil = 0;
    Serial.println("OK RESET");
  } else {
    Serial.println("ERR unknown");
  }
}
 
void loop() {
  // read characters until a newline, without blocking the rest of the loop
  while (Serial.available() > 0) {
    char c = Serial.read();
    if (c == '\n') {
      handle(line);
      line = "";
    } else if (line.length() < 40) {
      line += c;
    }
  }
  // switch the alarm off when its time is up (no delay(), so we keep listening)
  if (alarmUntil != 0 && millis() >= alarmUntil) {
    digitalWrite(LED_PIN, LOW);
    digitalWrite(BUZZER_PIN, LOW);
    alarmUntil = 0;
  }
}

Open the Serial Monitor at 115200 baud, line ending "Newline". Type PING and expect PONG. Type GOAL: the LED and buzzer come on for 1.5 seconds. The board should be working before any Python is involved. Close the Serial Monitor afterwards, because only one program can use the port at a time.

Cell 2: `hardware.py` (the Python side)

"""Talk to the goal board over a serial port. The automation must NOT crash if the board is missing."""
import time
 
 
class Hardware:
    """port: anything with write(bytes) and readline() -> bytes  (pyserial's Serial does).
    Timeouts are the port's job: open it with timeout=0.5."""
 
    def __init__(self, port=None, retries=1):
        self.port = port
        self.retries = retries
        self.ok = port is not None
        self.log = []
 
    def _note(self, msg):
        self.log.append(msg)
        print("[hardware]", msg)
 
    def send(self, command):
        """Send one command, wait for one reply line. Returns the reply text, or None if it failed."""
        if not self.ok:
            self._note(f"no board, skipped {command!r}")
            return None
        for attempt in range(self.retries + 1):
            try:
                self.port.write((command + "\n").encode())
                reply = self.port.readline().decode(errors="replace").strip()
            except OSError as e:                               # unplugged cable, port closed
                self._note(f"serial error: {e}. Carrying on without the board.")
                self.ok = False
                return None
            if reply:
                if reply.startswith("ERR"):
                    self._note(f"board did not understand {command!r}: {reply}")
                return reply
            self._note(f"no answer to {command!r} (attempt {attempt + 1})")
        return None
 
    def ping(self):
        return self.send("PING") == "PONG"
 
    def on_goal(self, zone, t):                                 # handler for EventBus.on("goal", ...)
        self.send("GOAL")
 
    def show_score(self, score):
        self.send(f"SCORE {int(score)}")
 
    def reset(self):
        self.send("RESET")
 
 
def open_board(name=None, baud=115200):
    """Find and open the real board with pyserial. Returns a Hardware that works even if nothing is found."""
    try:
        import serial
        from serial.tools import list_ports
    except ImportError:
        print("pyserial is not installed (pip install pyserial). Running without hardware.")
        return Hardware(None)
    if name is None:
        ports = list(list_ports.comports())
        if not ports:
            print("No serial ports found. Running without hardware.")
            return Hardware(None)
        name = ports[0].device                                  # simple choice: the first one. Print them all if unsure.
        print("Available ports:", [p.device for p in ports], "-> using", name)
    try:
        port = serial.Serial(name, baud, timeout=0.5)
    except OSError as e:
        print("Could not open", name, e)
        return Hardware(None)
    time.sleep(2)                                               # many Arduinos reset when the port opens: wait for the start-up
    port.reset_input_buffer()
    return Hardware(port)

Cell 3: a pretend board

It follows the same rules as the sketch, so you can test the Python without any hardware.

"""A pretend goal board that follows the same rules as goal_board.ino, so we can test without hardware."""
 
 
class FakeBoard:
    def __init__(self, silent=False, unplugged_after=None):
        self.replies, self.silent = [], silent
        self.unplugged_after, self.count = unplugged_after, 0
        self.led, self.buzzer, self.blinks, self.history = False, False, 0, []
 
    def write(self, data):
        self.count += 1
        if self.unplugged_after is not None and self.count > self.unplugged_after:
            raise OSError("device disconnected")
        for raw in data.decode().split("\n"):
            if not raw:
                continue
            self.history.append(raw)
            cmd = raw.strip()
            if self.silent:
                continue
            if cmd == "PING":
                self.replies.append("PONG")
            elif cmd == "GOAL":
                self.led = self.buzzer = True
                self.replies.append("OK GOAL")
            elif cmd.startswith("SCORE "):
                n = int(cmd[6:] or 0)
                self.blinks += n
                self.replies.append(f"OK SCORE {n}")
            elif cmd == "RESET":
                self.led = self.buzzer = False
                self.replies.append("OK RESET")
            else:
                self.replies.append("ERR unknown")
 
    def readline(self):
        return (self.replies.pop(0) + "\r\n").encode() if self.replies else b""

Cell 4: tests (no hardware)

Copy events.py, goal_detector.py (lesson 31) and detections.json (lesson 30) into the folder.

from events import EventBus
from fake_board import FakeBoard
from goal_detector import GoalDetector
from hardware import Hardware
import json
 
bad = 0
def check(name, ok, extra=""):
    global bad; bad += not ok; print("PASS" if ok else "FAIL", name, extra)
 
b = FakeBoard(); hw = Hardware(b)
check("ping", hw.ping())
check("goal answered OK", hw.send("GOAL") == "OK GOAL" and b.led and b.buzzer)
check("score", hw.send("SCORE 3") == "OK SCORE 3" and b.blinks == 3)
check("reset", hw.send("RESET") == "OK RESET" and not b.led)
check("unknown command", hw.send("DANCE") == "ERR unknown")
 
hw2 = Hardware(FakeBoard(silent=True), retries=1)
check("silent board returns None after retries", hw2.send("GOAL") is None and len(hw2.log) == 2, hw2.log)
check("ok stays true for a silent board (it may just be slow)", hw2.ok)
 
hw3 = Hardware(FakeBoard(unplugged_after=1))
hw3.send("PING"); r = hw3.send("GOAL")
check("unplugged mid-match: no crash, switches to no-board mode", r is None and hw3.ok is False)
check("later commands are skipped quietly", hw3.send("GOAL") is None and "no board" in hw3.log[-1])
 
hw4 = Hardware(None)
check("no board at all", hw4.send("GOAL") is None and hw4.ping() is False)
 
# full chain: lesson 30 data -> lesson 31 detector -> bus -> hardware
data = json.load(open("detections.json")); b = FakeBoard(); hw = Hardware(b)
bus = EventBus(); score = [0]
def goal(zone, t):
    score[0] += 1; hw.on_goal(zone, t); hw.show_score(score[0])
bus.on("goal", goal)
det = GoalDetector(data["goal_zone"], bus)
for fr in data["frames"]:
    det.update(fr["ball"], fr["t"])
check("3 goals reach the board", b.history.count("GOAL") == 3 and b.history[-1] == "SCORE 3", b.history)
check("board blinked 1+2+3 = 6 times", b.blinks == 6, b.blinks)
print("failures:", bad)

All should print PASS. Read the three failure cases: a silent board, an unplugged cable mid-match, and no board at all. The automation never crashes.

Cell 5: the real thing

"""Replay lesson 30's detections and let goals light the real board. Run: python run_with_board.py [COM3 or /dev/ttyUSB0]"""
import json
import sys
 
from events import EventBus
from goal_detector import GoalDetector
from hardware import Hardware, open_board
 
 
def run(hw, detections_path="detections.json"):
    data = json.load(open(detections_path))
    bus, score = EventBus(), [0]
 
    def on_goal(zone, t):
        score[0] += 1
        print(f"GOAL at {t:.1f} s -> score {score[0]}")
        hw.on_goal(zone, t)
        hw.show_score(score[0])
 
    bus.on("goal", on_goal)
    detector = GoalDetector(data["goal_zone"], bus)
    hw.reset()
    for fr in data["frames"]:
        detector.update(fr["ball"], fr["t"])
    return score[0]
 
 
if __name__ == "__main__":
    board = open_board(sys.argv[1] if len(sys.argv) > 1 else None)
    print("board answers ping:", board.ping())
    print("goals:", run(board))

Run python run_with_board.py (it picks the first serial port) or python run_with_board.py COM3 (Windows) or python run_with_board.py /dev/ttyUSB0 (Linux). Three goals in the replay should give three beeps and the score blinks.

Questions: why does the sketch answer before it blinks? Why does the Python wait two seconds after opening the port? What happens if you press the board's reset button mid-match? How would you drive the same board from the Flask server in lesson 34?

If it goes wrong

Error	Cause	Fix
No port appears	Charge-only cable, missing driver	Try another cable, install the driver, check the Device Manager or `ls /dev/tty*`
`PermissionError` / access denied	Another program (Serial Monitor, Arduino IDE) has the port	Close it. On Linux, add yourself to `dialout`.
First command lost	Board resets when the port opens	Wait two seconds after opening; clear the buffer
No reply	Wrong baud rate or wrong port	Match 115200 in both; check the port name
Garbled text	Baud mismatch	Same baud rate in Serial.begin and Python
LED always on	Wrong wiring (anode and cathode swapped, or no resistor)	Long leg to the pin through the resistor
Buzzer is silent on ESP32	3.3 V is too low for some buzzers	Use a 3.3 V buzzer or a transistor (teacher-led)
Alarm never switches off	Sketch uses `delay()` or the time check is missing	Use the `millis()` check from the sketch

Try it

Add a different sound pattern for the second goal
Add a STATUS command that returns the alarm state
Send the same commands over Wi-Fi from an ESP32 (extension)

Check what you learnt

What is a protocol?
Why should the board acknowledge every command?
Why not use delay() for the alarm in the sketch?
Someone unplugs the cable mid-match. What should happen?
Why can't the Serial Monitor and Python use the port at the same time?

Conclusion

Computer vision can drive real hardware. Python sends short text commands over serial, the board replies OK, and both follow an agreed protocol. The sketch uses millis() instead of delay() so it keeps listening, and Python retries, then carries on gracefully if the board disappears.

What next?

Next: Automation stress test. You'll stress-test your whole system before trusting it at a live event.