Metal Fish Eggs

With the introduction of FLARToolkit and other means of tracking user input through a webcam in flash, I recently had a chance to read up on some of the other ways we can interact with flash applications using a webcam.

Camera.activityLevel

The most basic way to detect activity through a webcam in flash is by accessing the Camera object’s activityLevel property. This returns a number from 0 to 100, depending on how much activity is going on. While useful in simple applications, this doesn’t give us a way to check for specific gestures or a way to check a specific section of the video for activity.

BitmapData.getPixel()

A popular way to detect motion is by using the BitmapData object in flash. By taking “snapshots” of the video coming in from the webcam constantly (using either Event.ENTER_FRAME, or the Timer object), we end up with frames that we can compare by looping through each pixel and checking to see which has changed. This technique is useful for checking which areas of the video has activity. We can use it to interact with buttons or other display objects by simply waving our hand over it. Though it can be very intensive (specially when working with video that has large dimensions), it is one of the best and most reliable way to detect motion in flash.

Augmented Reality & Marker Tracking

As mentioned earlier, another option is to use saqoosha’s FLARToolkit. If you haven’t played with it before, you should definitely check it out…after reading my blog post to the end, of course. From what I understand (and correct me if I’m wrong), this neat library also uses the BitmapData object to take stills of the video from the webcam. It then manipulates the bitmaps in order to detect and extract the position of a marker in 3D space. You can then use flash 3D libraries like Papervision to apply those coordinates to a 3D object and display it on top of the marker as if it’s a physical object. This method requires the user to print out a specific symbol and display it in front of a webcam. The wicked thing about FLARToolkit is that it doesn’t restrict you to using just Papervision, you can use Away3D or no 3D at all!

Though it comes packaged with Papervision and Away3D, with a bit of “code spelunking”, you can strip out the 3D stuff and gain access to the marker’s x,y, and z properties. By doing this, you end up with a basic marker tracking library which you can use for pretty much anything your nerdy heart desires. In my case, my nerdy heart and I created a very basic game where you have to catch falling apples using only simple vector MovieClips. Check it out below:

Download Source

Moving MovieClips with your hand (or other limbs) using the webcam

Now after playing with the BitmapData motion detection and the FLARToolkit, I went on a mission. A mission to find/figure out a way to interact with display objects on the stage using the webcam, BUT without using markers. After a bit of searching, I was able to find this demo on youtube which is close to what I want but with more control over the display object’s reaction.

Using what I’ve learned, and some inspiration from the youtube demo, I threw a quick example together as a learning exercise for myself. By checking for activity where our “box” movieclip is located, we can check which side has the most activity and move the box in the proper direction as if it’s a physical box you can push around. Though it’s not perfect and could DEFINITELY use some more code optimization, I was very satisfied with the result.

Download Source

AS3 Real-Time Face Detection

Another technique worth mentioning is face detection in flash. Here’s a snippet taken from Mario Klingemann’s blog about the subject:

“Yesterday Seb-Lee Delisle pointed us via Twitter to this great example of a Flash based real-time face detection. It turns out that already more than half a year ago Ohtsuka Masakazu had been porting the face detection part of OpenCV to AS3 and added the source code to the Spark project – which is like a Actionscript candy box full of surprises.”

He then explains how it works and how to optimize it, it’s a good read for those interested in the subject.

So there you have it, we went from detecting basic general webcam activity in flash to complex face detection algorithms. Big kudos to the folks mentioned in this post as they’ve put in many hours coding in order for the rest of us to create awesome and engaging content with ease.