Example inspired by the “Awesome digital bokeh effect in Photoshop” tutorial by Abduzeedo.

Click here to view the embedded video.

Used tools:

• Gl Graphics 1.0
• Processing 1.5.1

Please note that Gl Graphics doesn’t work in Processing 2.0 beta yet.

For explanation see comments in the code.

Dowload BokehParticleSystem code

Send and receive OSC (Open Sound Control) messages with Scratch. You can interface Scratch with external sensors and OSC sending applications/apps. You can also use Scratch to send OSC data to control audio software, VJ software or creative applications.

The bridge converts OSC messages to Scratch sensor and broadcast messages. More technical information about this you’ll find at: http://wiki.scratch.mit.edu/wiki/Remote_Sensors_Protocol

The application was build in Processing. You can download the application for OSX, Windows and Linux, a Processing Sketch is also available.

Download Scratch OSC Bridge from Google code

The downloads include some Scratch example projects with code commented. I’ve also implemented a smoothing algorithm to make movement more fluid.

Scratch TUIO

Move the Scratch cat with a TUIO client.

Good TUIO trackers:

• tuioDroid (tracker for Android)
• tuioPad (tracker for iOS devices
• tongseng (use macbook trackpad as tuio tracker)

More TUIO tracker implementations at tuio.org

Scratch TouchOSC

Click here to view the embedded video.

Use the acceleration sensor data to move the cat. Change the color with fader5 and change the background with different layouts.

• touchOSC
• touchOSC for Android
• IP address configuration (Scratch OSC Bridge displays the IP in the left upper-corner)

Scratch Bridge configuration

Configure the Scratch OSC Bridge with broadcast messages from Scratch. Instead of setting everything when starting the client, you can do this with broadcast messages from Scratch.

More info on OSC

the osc protocol osc tutorial

Download: Adobe After Effects Cheat Sheet

In the example below you’ll see two balls that change colour and position when hit. Code is in the zip-file.

See my post about Frame differencing for other Flash and Processing examples.

Example (Flash Player 10 or higher)

You need a webcam and give Flashplayer access to the camera to see the example below :

Used resources

• Using BitmapData.hitTest for Collision Detection by Mike Chambers
• code snippet from Zevan

Download : Framedifferencing collision detection – source code

If you have suggestions for more improvements, please give your feedback in the comments.

A normal Flickr RSS feed will give you only the 20 most recent pictures (and only in thumb or original format) while we needed all the pictures of a whole week. Thanks to a sample on Yahoo Pipes I could create a feed with an unlimited amount of pictures and with the format of 500×500 (sufficient resolution for our demands. Unfortunately Yahoo Pipes isn’t very stable, so in the end we used a downloaded (not realtime) version of the RSS feed as a backup.

Download the composition: Flickr Polaroid stream

The code is pretty well documented, the graphic design of the polaroids needs some improvement. Let my know in the comments if you used this composition or made an alternation on it.

A video of the Students in Motion projection mapping in Amsterdam (thanks to MediaLAB Amsterdam and Escapation).

Click here to view the embedded video.

Some links to help you get started with Quartz Composer (included in the developer tools for Mac, so its free when you own a mac).

• Quartz composer userguide
• Quartzcompositions.com
• Kineme.net
• Quartzcomposer.com

Steps:

• Download the library : Arduino Moodlight library
• Unpack the library in the Arduino libraries folder (standard in documents, arduino).
• Restart Arduino if it was running.
• Open the moodlight file in File > Examples > MoodLight > moodlight.

You can see and use the original code in this example: Arduino Programming – HSB to RGB

You can find more information about using Arduino libraries on the Arduino Environment page.

Moodlight.pde :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
// Include the MoodLight library.

#include <MoodLight.h>



// inputs

int sensorPin = 0;



// outputs

int ledPinR = 9;  // pwm pin with red led

int ledPinG = 10; // pwm pin with green led

int ledPinB = 11; // pwm pin with blue led



// other variables

int sensorVal;   // store value from sensorPin



int hue;         // use value between 0 - 359

int saturation;  // use value between 0 - 255

int brightness;  // use value between 0 - 255



// create MoodLight object

MoodLight ml = MoodLight(); 



void setup() {  

  

   pinMode(ledPinR, OUTPUT);  

   pinMode(ledPinG, OUTPUT);  

   pinMode(ledPinB, OUTPUT);   

   

}



void loop() { 

  

  // read the sensor (potmeter)

  sensorVal = analogRead(sensorPin); 

  

  // map the value from 0-1023 to a value between 0 and 359

  hue        = map(sensorVal,0, 1023,0, 359);

  saturation = 255; 

  brightness = 255; 

  

  // set the Moodlight values  

  ml.setHSB(hue, saturation , brightness);



  // read the Red, Green and Blue values

  // and write them to the led pins

  analogWrite(ledPinR, ml.getRed());

  analogWrite(ledPinG, ml.getGreen());

  analogWrite(ledPinB, ml.getBlue());  

  

}

You can use this formula in every programming/scripting language of course. Below you can download a more complex example for Scratch4Arduino (S4A) that I use in my classes (it also includes a smoothing formula).

s4a-mapping-smoothing-v3.zip

The Arduino and Processing languages already have an implementation of this formula in the map() function. For an example see this post: Arduino Programming – Map and smooth values

Modifications

• Implemented Adaptive Threshold by Quasimondo (Mario Klingemann) in the GetQRimage Class.
• Added the substract blur method that Mario Klingemann explains in his lecture about the process of reading QR codes. I’ve got less wrong strings in this mode and the detection runs faster. You can set it to false, to use the original method.
• The outside parts of the video are cropped, to reduce image noise. By providing a markerGuide the user knows where to hold the marker.
• The data from the onQrDecodeComplete event is compared with the data from a previous event as a way of extra error correction.

The user interface uses the minimalcomps by Keith Peters (bit-101) and the Hi-Res! stats by Mr. Doob.

Example (Flash Player 10 or higher)

Generate a QRcode and print it out.
You need a webcam and give Flashplayer access to the camera to see the example below :

Used resources

• Introduction of QR Code Reader Library
• QRCodeReader library (Libspark)
• Max 2008 develop – Be here pixels by Mario Klingemann (interesting lecture)
• Automated Threshold & Edge Detection

Download : ReadQrSimple – source code

If you have suggestions for more improvements, please give your feedback in the comments.

Since the 4 years my colleagues and I teach Physical Computing with Arduino and Flash (see my tutorials on Flash Arduino communication). The students are not allowed to make use of standard keyboard/mouse interaction, so they used Arduino in all of the projects. Apart from that some students used computer vision as well as input. Each team gets a laptop to use in the project, but some take the challenge to make a complex installation running on Arduino only.

This year 24 groups did their presentation. The level was really high and they’ve made some really good installations. Its a touch job to make something that functions good in hard- and software.

My three personal favorites below.

Kosmic Kitchen – Digital Praxinoscope

A modern tribute to a historic invention. The Praxinoscope is a installation of 16 mirrors on a spinning wheel, when spinning it shows the 16 frames and you’ll get an illusion of movement. You can select animations and background colors with an old fashioned telephone dialer. The group made a beautiful looking and good working installation that fits perfect in the shop.

Hardware – software : Arduino Mega, 4 RGB leds, telephone dealer, record player, microphone, Sharp distance sensor, Flash, Arduino AS3 Messenger.

Comicasa – Welke actie held ben jij ? (Which action hero fits you ?)

In a few questions, a fingerprint scan and a face scan this installation detects which superhero fits you the best. The phone rings to attend you that you can see your superhero combination on the website and on Twitter. The installation is placed in the Comic book and game store Comicasa. The installation works perfect and uses the curiosity of people find out who fits them and of course the vanity of their picture on the web. Its designed like a console and they made smart use of the laser cutter to cut the frames and logo.

Hardware – software : Arduino, Switches, Pressure sensor, PIR, Ledstrip, servo motor, Flash, PHP, MySQL, As3Glue.

Bagels and Beans: Bagel Balance Board

Balancing on a seesaw you have to put together your favorite bagel. You’ll have to gather a maximum of 5 ingredients and if you succeeded a ticket is printed with the ingredients for a discount price. A simple game that works out great with the seesaw balancing board. The installation integrates nice in the environment of the shop and the combination of a discount ticket for a bagel is really motivating for the players.

Click here to view the embedded video.

Hardware – software : Arduino, Sharp distance sensor (seesaw), buttons, leds, Flash, As3Glue.

De Leckernij: Cheezy

A old fashioned game where you have to feed “Boris Slagmuis” a piece of cheese. With 2 buttons you have to move a ball to the right hole. I nice hardware only installation which uses 2 Arduino’s, a Vmusic, two servo’s and different sensors. Nice design and great use of electronics.

Hardware – software : Arduino (2x), Flex sensor, PIR, LDR, arcade buttons, leds, Vmusic.

U-turm

An interactive projection for the closing ceremony of the E-culture fair, as part of the cultural capital of Europe Ruhr 2010. Dance on the facade of the Dortmunder U. In collaboration with producer Jan Scholte and the artist Matthias Oostrik.

The U-turm team made an interactive projection on the Dortmunder-U tower. When a person steps on one of the stages their silhouette appears on one of the windows of the building. Depending on how the person moves the silhouette can move to other windows, leaving a trace where it has already been before. If two or more silhouettes meet on the same window there will be some creative content coming out, symbolizing the new use of the building as cultural hotspot.

The installation uses Quart Composer the generate the visuals. I’ve helped Joost Buijs (responsible for the technical part) finding some different motion tracking solutions for Quartz Composer.

Tested options were :

• v002 Optical Flow 2.0.2 : uses a lot of processing power, so a big framerate drop.
• Webcam piano by Memo Akten : worked good, but didn’t invite for dancing too much.
• Kineme CVTools plugin : used in the end, saves a lot of processing power by checking only certain points.

The team did a good job to find and test the right way of interaction. Its a really powerful and intuitive concept I’m curious how its going to look on the big building.

Click here to view the embedded video.

Scryption – Google mirror.

The team of Scryption made a mirror that shows your digital identity. The concept was based on the theories of Marshall McLuhans ideas. From their concept document :

You enter the area and you will see two different mirrors in front of you. You will be directed to type in your name. The left mirror is just an ordinary mirror that shows the real you. The left one shows up the pictures that appear on Google when doing a search query on your name. This mirror shows you the digital you, the identity that is created by the internet (incl. other people) and by yourself. You will then be able to play with your digital shape since there are sensors installed that react on your movement. A picture is uploaded to the net.

The installation fits to the debate around technological determinism. As McLuhan puts it, “We shape our tools and our tools shape us”. But also that the medium is the message since a mirror always confronts you with yourself. And sometimes you see something you didn’t expect to see.

Saro van Cleynenbreugel was responsible for the technical part. Together with him and project coach Iris Douma we’ve decided to make the installation with Processing. I like Processing because of the ease of code sharing (mail some code and run it directly). However in the end Processing seemed a bit to slow for the task. Especially blending a camera silhouette with images and text cost a lot frames. Maybe next time I would choose and suggest OpenFrameWorks for an installation like this. In the end the application ran at 20 fps, so we couldn’t complain. Due some problems with a Nightvision camera (warning: exposure settings don’t work in that mode), there was some trouble to make a nice looking silhouette.

Libraries used :

• Fullscreen library
• OpenCV library
• Box2D library

A lot of interactive tools like Flash and Processing serve well to make visual effects. By using relative simple scripts ( re-use and play ! ) you can create complex and cool looking visuals. Most tools let you export the material to image sequences or even video. Make use of them even for non-interactive stuff. Try some examples to create some genenerative art yourself.Lets start using your creativity and start playing, ‘remixing’ and mashing up some code.

Toolkits :

• processing.org
• quartz composer (osx)
• hype framework (Flash)
• Nodebox (Python, nodebased)
• vvvv (nodebased, windows)
• Tagtool/Nodekit (windows)
• after effects expressions
• OpenFrameworks (c++)
• Cinder framework (c++)

Code examples, libraries, inspiration :

• sc/4 : realtime compositing by Minivegas
• memo.tv
• kineme.net (quartz)
• levitated : Jared Tarbell
• Joshua Davis
• incubator : Mario Klingemann
• toxic libs ( processing libraries examples
• processing learning section
• cinema redux : Bredan Dawes (2004)
• yooouuutuuube.com : David Kraftsow 2009 | Alice ( Fagottron )

Books :

Above 20 euro no shipping costs to the Netherlands and Germany !

For receiving of data on the Arduino I’ll choose the Messenger library. It has some nice simple functions and gives you the opportunity to check if a string is fully received (a ‘carriage return’ is send by Flash at the end of a serial message in the SerialPort class) by Arduino. Of course you can write your own serial protocol. You can find some thoughts about it at the Todbot blog : Arduino serial protocol design patterns.

You can send data to the serial port with the Serial.print() and Serial.println() commands.

Included is the SerialPort class for Flash that makes it easy to send and receive serial data from/to the Arduino. I’ve modified the original (from tinker.it) so it supports start and stop characters. As a stop character ‘carriage return’ is used (‘\r’, or char(13)) that is send when you do Serial.println(); from Arduino.

I advise using a start character (not used in the example below) as well, because sometimes (especially when sending on a high datarate) characters at the beginning if the string can get lost.

The download contains SerialPort.as and example_messenger.fla (CS3).

Download : flash-arduino-as3-messenger

Step by step guide for Messenger library communication

• Download the Messenger library and install it in your Arduino/libraries/ folder ( you can find it by default in your documents folder ). Restart Arduino if it was running.
• Open basic_communication_old in my zip-file (basic_communication included in the Messenger examples has been changed).
  I’ve modified the old example in two ways. I’ve changed the serial speed to a baudrate of 57600, so you don’t have to change the Serproxy.cfg if you use it with Firmata as well and I have change the update interval to 40ms (instead of 20ms) for more stability.
• Upload the Sketch to the Arduino.
• Configure Serialproxy for your Arduino board and run it.
• Open example_messenger.fla and run it.
• If everything went fine you’ll see the led on pin 13 blinking and 6 balls, controlled by the analog inputs, moving on the screen.

Example_messenger code

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
var receivedValues:Array = new Array();



// Arduino connection



/* SerialPort(localhost, netport, start character, stop character)

   for more stability use a startcharacter so you can check if a message is complete

   default stop character is carriage return '\r' 

   A carriage return is send by the Arduino command Serial.println();

*/



var arduino:SerialPort = new SerialPort("127.0.0.1", 5331);

arduino.addEventListener(DataEvent.DATA, onArduinoData );

arduino.connect();



// Setup a timer to send data to the Arduino

// The timer object calls the timerEvent function 20 times a second (every 50ms)

var timer = new Timer(500);

timer.addEventListener("timer", timerEvent);

timer.start();



// toggle variable that is switched by the timer from true to false and back

var toggle:Boolean= true;



function timerEvent(event:Event):void 

{   

    // change toggle boolean to the opposite

    toggle = !toggle;

    

    if(toggle == true)

    { // make ledPin 13 HIGH - onboard led on 

      //arduino.send("on"); // For Messenger checkString example

      arduino.send("13 1"); // For Messenger basic_communication example

    }

    else

    { // make ledPin 13 LOW - onboard led off

      //arduino.send("off"); // For Messenger checkString example

      arduino.send("13 0");  // For Messenger basic_communication example

    }

    

    /*

    When you have a certain amount of values it will be smart to keep them

    in an array (so you only modify the array indexes that change.

    

    You can convert an array to a character seperated string (space character for 

    Messenger library) with the join function.

    

    Example :

    var sendValues:Array = new Array(2,4,8,16,32,64,128);

    arduino.send(sendValues.join(" "));

    

    */

}



// add some balls to the stage to display incoming analog values

var ballArray = new Array();



for(var i:int = 0;i<6;i++)

{   

    var ball:MovieClip = new Ball();

    ballArray[i] = ball;

    ball.width = ball.height = 50;

    ball.y = 10;

    ball.x = 10+(i*60);

    addChild(ball);

}



// Called when data is received from the Arduino

function onArduinoData( event:DataEvent ):void

{   

    trace("received string : ", event.data );



    /* The values from Arduino Messenger are space seperated values. We will 

       store each value in the receivedValues array.    

    */

    receivedValues = event.data.split(" "); 

    

    // change the y position of the balls on stage 

    for(var i:int = 0;i<6;i++)

    { var mc = ballArray[i];

      mc.y = receivedValues[i] * 350/1023;  

    }   

}

Project: unity-arduino with serial connection

This project uses Unity (which uses C# and JavaScript scripting), the Arduino and/or SerialProxy to connect the Arduino to Unity. Thus allowing custom hardware to be linked to Unity (in real-time).

The link is established in Windows using the Mono implementation of System.IO.Ports.

For Mac/OSX users, I’ve made an implementation that uses SerialProxy because direct serial connection doesn’t work on the mac. The problem is, though, that the SerialProxy implementation doesn’t seem to work either.

Links:

• Unity-Arduino serial-connection on Google code
• Unity : Game Development Tool
• Arduino

Blob Detection test by samkeene :

Click here to view the embedded video.

Max Meinders made a good description of all the processing steps involved in Blob tracking :
Max Meinders – videotracking.

Face detection

OpenCV also supports the recognition of special characteristics in an image and use those as a blob. Face detection a tecnique to recognize things that look like a face in an image. It doesn’t recognize a certain face ( although thats possible, see Image recognition ), but it detects facial features.

Face Detection with Processing by Pedro Amado

Click here to view the embedded video.

Face detection works with classifiers. You can learn the computer characteristics of an object. So OpenCV can detect other objects as well, as long you have a classifier file for it.

Keywords : Haar Classifier, Viola Jones object detection framework, cascade classifiers.

Tomek made a nice example where he tracks faces in the ‘Black and White’ video of Michael Jackson in realtime in Flash ( you can tried it yourself to with a webcam ) : Machines are looking for Michael Jackson.

Below an example of the difference filter in the OpenCV library for Processing. You can do the same in Flash with the Difference Blendmode.

Click here to view the embedded video.

You can move objects to area’s where most of the motion is happening, like in the Pong example of Ostrich : Ostrich two cursors example

Or you can make some hotspots in your image and check the amount of motion in those. A nice example to get the idea is the webcam piano from Memo Akten (implemented in Quartz Composer and Processing) :

Click here to view the embedded video.

Or you can do other creative things with it (video by WTF? by OK Go, example code in Processing by Memo Akten) :

Click here to view the embedded video.

Code examples and tutorials

If you only want to detect if there is a certain amount of motion in front of a webcam, you can use the activityHandler and activityLevel parameters in Flash. See my example on bitmap filters where i use it the apply a red filter when there is movement :
Flash AS3 : Apply bitmap filters to a webcam image