Catch the stars: body pose detection

Introduction

Some video games let you play with your whole body: you jump, duck and wave, and the character copies you. The camera is tracking your pose: where your head, shoulders, elbows, wrists, hips and knees are.

In this lesson you'll use PictoBlox's Human Body Detection blocks to follow your wrists, then build a game where you catch falling stars with your hands.

Working principle

• Analyse image for human pose from (camera) finds the people in the camera picture and their keypoints.
• Get # of people tells you how many people it found.
• The x position and y position blocks report where one keypoint is (for example the right wrist) for one person, in stage coordinates.
• Is (keypoint) of person (1) detected? is true when that body part is visible. If your hand is out of the picture, it's false.
• Your right wrist appears on the left of the stage if the video isn't mirrored, just like in a photo of you. Don't be surprised!

What you'll need

• PictoBlox with the Human Body Detection extension
• A webcam, and enough space to stand back so your upper body fits in the picture
• Two sprites: a Ball (it will follow your wrist; make it small and see-through) and a Star

Working in real life

1. Add the Human Body Detection extension.
2. Make a small sprite called Hand (a ball works) and build script 1 on it.
3. Choose the Star sprite and build script 2 on it. Make a variable called score.
4. Stand back from the camera so it can see you from the waist up, click the green flag and catch the stars.

Code

Script 1, on the Hand sprite:

when green flag clicked
turn [on] video on stage with (30) % transparency
forever
  analyse image for human pose from [camera]
  if <is [right wrist] of person (1) detected?> then
    show
    go to x: (x position of [right wrist] of person (1)) y: (y position of [right wrist] of person (1))
  else
    hide
  end
end

Script 2, on the Star sprite:

when green flag clicked
set [score v] to (0)
forever
  go to x: (pick random (-200) to (200)) y: (180)
  repeat until <(y position) < (-170)>
    change y by (-4)
    if <touching [Hand]?> then
      start sound [Collect]
      change [score v] by (1)
      go to x: (0) y: (-200)
    end
  end
end

The exact block text can differ a little between PictoBlox versions. Look in the Human Body Detection palette for the blocks that report a keypoint's x and y position.

Try it

• Add a second Hand sprite that follows your left wrist, so you can catch with both hands.
• Add a Rock sprite. If you touch it, you lose a point!
• Make a "raise your hand" detector: if the right wrist's y is higher than the nose's y, Tobi says "Yes, you have a question?"

Conclusion

• A pose model finds keypoints on the body and gives you each one's position.
• Always check a keypoint is detected before you use its position.
• Comparing keypoints (wrist above nose) lets you detect actions like a raised hand.

What next?

Now zoom in on your hand and draw in the air: Draw in the air: hand tracking.