Week 3

06MA4010991AP
Computer Game Design

Week 3

Optical motion detection by image processing

Obtain globs from frame difference

The above images are captured from two consecutive frames from webcam, there are slightly difference between them as where the motion is. To locate the difference part clearly, we can perform subtraction between each pair of pixels.

The moving parts are shown in color and the rest appeared as black. Then we can do threasholding on the brightness of the resulting pixels, and get a clearer indication.

Afterwards, we can group the nearby white pixels to form globs. There may exist many small globs which are possibly caused by camera noise, so they'll be ignored at most of the cases.

JMyron

JMyron is a processing library which do the above job for us, it is a handy tool to perform simple computer vision. To setup, we visit http://webcamxtra.sourceforge.net/ and go to the download page.

Click on the Download JMyron 0025 to obtain the latest version, unzip and follow the instruction inside HowToInstall.txt. It's simply copy the "JMyron" folder found in the directory into the Processing/libraries directory. If you are using windows, there is a folder called "Extra DLLs" - those are DLLs that JMyron depends on so make sure they're copied into your system's DLL search path - such as the Processing root directory, or even C:\windows\System32.

Example 1

Source

The follow example is an introduction as how to use the data fetched in JMyron library.

import JMyron.*; //import JMyron library

JMyron cam; //camera object
PImage output; //for displaying camera image
int mode; //changing between different modes

void setup()
{
size(320,240);
cam = new JMyron(); //initialize jmyron
cam.start(320,240); //set the camera resolution
cam.findGlobs(1); //ask jmyron to start finding globs
cam.adaptivity(50.0); //duration of frame buffer
output = new PImage(320,240); //initialize output image reoslution

mode = 0;
smooth();
}

void draw()
{
background(0);
cam.update(); //update camera image
output.pixels = cam.cameraImage(); //fill camera data into output
output.updatePixels(); //update output image
image(output,0,0); //display output image

if(mode == 0)
{
//glob centers
int[][] globs = cam.globCenters();
for(int i=1;i<globs.length;i++)
{
ellipse(globs[i][0],globs[i][1],5,5); //draw ellipse on each glob center
}
}
else if(mode == 1)
{
//glob boxes
int[][] globs = cam.globBoxes();
for(int i=1;i<globs.length;i++)
{
rect(globs[i][0],globs[i][1],globs[i][2],globs[i][3]); //draw rect on each glob
}
}
else if(mode == 2)
{

//glob edge points
int[][][] globs = cam.globEdgePoints(5);
for(int i=1;i<globs.length;i++)
{
beginShape();
if(globs[i] != null)
{
for(int j=0;j<globs[i].length;j++)
{
vertex(globs[i][j][0],globs[i][j][1]); //draw shape of each glob
}
}
endShape(CLOSE);
}
}
else if(mode == 3)
{
//glob quads
int[][] globs = cam.globQuads(3,5);
for(int i=1;i<globs.length;i++)
{
//draw simplified shape(quad) of each glob
beginShape();
vertex(globs[i][0],globs[i][1]);
vertex(globs[i][2],globs[i][3]);
vertex(globs[i][4],globs[i][5]);
vertex(globs[i][6],globs[i][7]);
endShape(CLOSE);
}
}
}

void keyPressed()
{
//switch to next mode
mode++;
mode%=4;
}

JMyron starts with a camera, so it does the job to handle and fetching data form a webcam. After each frame it starts to analyze the globs and the above are 4 kinds of data to get in JMyron.

1) Glob centers

Glob center is the center position (in x,y) of each glob. JMyron has a globCenters() function to obtain the values, and it represents as a 2D array:

int[][2] : [[x,y],[x,y],[x,y]...]

2) Glob rectangles

Glob rectangle is the surrounding rectangle of each glob, it is aligned to x and y axis and tells the dimensions of the glob. JMyron has a globBoxes() function to obtain the values, and it represents as a 2D array:

int[][4] : [[x,y,w,h],[x,y,w,h],[x,y,w,h]...]

3) Glob edges

Glob edge is the surrounding edge of the glob, it tells the shape of the glob. JMyron has a globEdgePoints() function to obtain the values, and it represents as a 3D array:

int[][][2] : [[[x,y],[x,y],[x,y]...],[[x,y],[x,y],[x,y]...],[[x,y],[x,y],[x,y]...]...]

4) Glob quadrilaterals

Glob quad is simplified version of the surrounding edge of the glob, it only have 4 points. It can tell the direction of the shape if it is long. JMyron has a globQuads() function to obtain the values, and it represents as a 2D array:

int[][2] : [[x1,y2,x2,y2,x3,y3,x4,y4],[x1,y2,x2,y2,x3,y3,x4,y4],[x1,y2,x2,y2,x3,y3,x4,y4]...]

Example 2

Source

The below example just simply make use of the above shapes as brushes and to draw on canvas.

import JMyron.*;

JMyron cam;
PImage output;
int mode;

void setup()
{
size(640,240);
background(0);
cam = new JMyron();
cam.start(320,240);
cam.findGlobs(1);
cam.adaptivity(50.0);
output = new PImage(320,240);
mode = 0;
smooth();
}

void draw()
{
//no background() to clean up

//flip image
translate(width,0);
scale(-1,1);

cam.update();
output.pixels = cam.cameraImage();
output.updatePixels();
image(output,320,0);

colorMode(HSB);
fill(frameCount%255,255,255); //loop in vivid colors

if(mode == 0)
{
//glob centers
int[][] globs = cam.globCenters();
for(int i=1;i<globs.length;i++)
{
ellipse(globs[i][0],globs[i][1],5,5);
}
}
else if(mode == 1)
{
//glob boxes
int[][] globs = cam.globBoxes();
for(int i=1;i<globs.length;i++)
{
rect(globs[i][0],globs[i][1],globs[i][2],globs[i][3]);
}
}
else if(mode == 2)
{
//glob edge points
int[][][] globs = cam.globEdgePoints(5);
for(int i=1;i<globs.length;i++)
{
beginShape();
if(globs[i] != null)
{
for(int j=0;j<globs[i].length;j++)
{
vertex(globs[i][j][0],globs[i][j][1]);
}
}
endShape(CLOSE);
}
}
else if(mode == 3)
{
int[][] globs = cam.globQuads(3,5);
for(int i=1;i<globs.length;i++)
{
beginShape();
vertex(globs[i][0],globs[i][1]);
vertex(globs[i][2],globs[i][3]);
vertex(globs[i][4],globs[i][5]);
vertex(globs[i][6],globs[i][7]);
endShape(CLOSE);
}
}
}

void keyPressed()
{
mode++;
mode%=4;
}

Flipping left and right in processing can be done at this way, make use of scale and translate function.

1) Translate the target canvas to width
2) Scale the x dimension to -1

In terms of code is like below:
translate(width,0);
scale(-1,1);

Continuous color can be done when using HSB color space easily, since HSB color space describe colors in cylinder shape where RGB space describe in cube shape.

(pics from wiki)

while holding satuation and brightness at 100% we can obtain color by changing it's hue, and the beginning and the end of the spectrum is the same. So that we can repeat the pattern to obtain a continuous color pattern.

in terms of code:

int i=0;

void draw()
{
colorMode(HSB);
i++;
color c = color(i%255,255,255);
}

Example 3

Source

Lets apply this input onto the game on week1!

import JMyron.*;

JMyron cam;
PImage output;

float mobX,mobY;
float mobSize;
int score;
int time;

PFont font;

void setup()
{
size(320,240);
background(0);
cam = new JMyron();
cam.start(320,240);
cam.findGlobs(1);
cam.adaptivity(50.0);
output = new PImage(320,240);
smooth();
font = loadFont("font.vlw");
initVariable();
}

void initVariable()
{
mobX = random(0,width);
mobY = random(0,height);
mobSize = 40;
score = 0;
time = 1000;
}

void draw()
{
background(0);
if(time>0)
{
pushMatrix(); //start flipping
translate(width,0);
scale(-1,1);

cam.update();
output.pixels = cam.cameraImage();
output.updatePixels();
image(output,0,0);

//draw mob
fill(255,0,0);
rect(mobX-mobSize/2,mobY-mobSize/2,mobSize,mobSize);

int[][] globs = cam.globCenters();
//hit test each glob
for(int i=1;i<globs.length;i++)
{
if(globs[i][0]>mobX - mobSize/2 && globs[i][0]<mobX+mobSize/2 && globs[i][1]>mobY-mobSize/2 && globs[i][1]<mobY+mobSize/2)
{
score++;
mobX = random(0,width);
mobY = random(0,height);
}
}
popMatrix(); //stop flipping

time--;
fill(255);
textFont(font);
text("Time: "+time,20,40);
text("Score: "+score,20,80);
}
else
{
fill(255);
textFont(font);
text("Score: "+score,20,100);
text("Press any key",20,140);
text("to restart",20,180);
}
}

void keyPressed()
{
if(time<=0)
initVariable();
}

Problem:
I want to flip the camera image and the mob (red box), but not the text (score and time).

Solution:
Make use of push and pop matrix.

Superficie Attiva: Water Effect from outsidestandinglevel on Vimeo.

Archiphonic Project from Koki IBUKURO on Vimeo.

Submersed Songs | Canções Submersas from ∆LEX on Vimeo.

the.blob.track_HD from MOVOPE PROD. on Vimeo.