Draw in the air: hand tracking

Introduction

What if you could draw on the screen just by moving your finger in the air, like a magic wand? In this lesson you'll track your hand with the webcam and use PictoBlox's Pen to leave a trail wherever your finger goes.

Working principle

• Analyse image for hand from (camera) looks for a hand in the camera picture.
• Is hand detected? is true when a hand is in view.
• The hand model finds 21 keypoints on a hand: the wrist, and four points along each finger and the thumb. The fingertip of the index (pointing) finger is the one we want.
• The position blocks report a hand keypoint's x and y on the stage.
• The Pen extension draws a line wherever a sprite moves while the pen is down.

What you'll need

• PictoBlox with the Human Body Detection and Pen extensions
• A webcam and good light on your hand
• A small sprite to act as the pen tip (a Pencil or a small Ball)

Working in real life

1. Add the Human Body Detection and Pen extensions.
2. Choose a small sprite and build the script below on it.
3. Hold your hand up to the camera with your pointing finger out and move it slowly. Press space to clear your drawing.

Code

when green flag clicked
turn [on] video on stage with (40) % transparency
erase all
set pen color to [blue]
set pen size to (6)
forever
  analyse image for hand from [camera]
  if <is hand detected?> then
    go to x: (x position of [index finger tip] of hand) y: (y position of [index finger tip] of hand)
    pen down
  else
    pen up
  end
end
 
when [space] key pressed
erase all

When your hand leaves the picture the pen lifts, so you can start a new line somewhere else. In your version of PictoBlox the fingertip may be chosen from a dropdown of keypoints; pick the tip of the index finger.

Try it

• Press 1, 2 and 3 to change the pen colour.
• Make the pen thicker when your hand is closer to the camera. (Hint: compare the wrist and fingertip positions; the bigger the distance, the closer your hand.)
• Write your name in the air, then take a screenshot to keep it.

Conclusion

• Hand tracking finds 21 keypoints on your hand in every frame.
• Moving a sprite to your fingertip with the pen down lets you draw in the air.
• Lifting the pen when the hand disappears keeps the drawing neat.

What next?

Faces, bodies and hands are done. Now let's find everyday objects: Treasure hunt: object detection.