06MA4010991AP
Computer Game Design
Week 4
Problem solving skills
In this chapter, we will go through some basic concept on problem solving skills. It is essential for developing any application.
Step 1 - What is your task (goal)?
The first step is to clarify your task, it makes you understand your task and easy to point out the difficulties / problems.
Lets create a face-detecting photo booth as class exercise.
Lets list out the requirements to develop such an application:
1) Obtain face position and size
2) Put the props in correct position and size
Step 2 - Break down your tasks into smaller piece
This step helps you to find a way to achieve your goal, or find out that is no possible way to achieve the task.
Broken down tasks:
Obtain face position and size
1) Look for tools which can do face detection
2) Install the necessary parts
3) Test with example
Put the props in correct position and size
1) Look for tools to control position and size of image
2) Deal with alignment problems
Step 3 - Tackle the tasks
Visit http://processing.org/reference/libraries/index.html you may be able to find libraries which can perform your task.
Here we found two libraries to deal with face detection depends on your platform. Lets tryout the PC one by Bryan Chung:
In our lesson we use processing 0135, so get the "Last one". Install that as most of the other libraries by putting into "libraries" folder. Face detection library also requires some .dll files from openCV in root of the processing folder and here is a link to download:
After the setup of library, lets try out the provided example (copy from web)
Version 1
import pFaceDetect.*; //import facedetect library import JMyron.*; //import facedetect use jmyron to get camera PFaceDetect face; //face detector JMyron m; //jmyron camera PImage img; //output pixel void setup() { size(320,240); m = new JMyron(); //init jmyron m.start(width,height); //init camera m.findGlobs(0); //switch off glob detection face = new PFaceDetect(this,width,height, "haarcascade_frontalface_default.xml"); //init face detector to recognize "frontalface" frameRate(15); img = createImage(width,height,ARGB); rectMode(CORNER); noFill(); stroke(255,0,0); smooth(); } void draw() { background(0); m.update(); //update camera arraycopy(m.cameraImage(),img.pixels); //update output image img.updatePixels(); //update output image face.findFaces(img); //find faces image(img,0,0); //draw cam image drawFace(); //draw faces } void drawFace() { int [][] res = face.getFaces(); //obtain faces data if (res.length>0) { for (int i=0;i<res.length;i++) { int x = res[i][0]; //face x int y = res[i][1]; //face y int w = res[i][2]; //face width int h = res[i][3]; //face height rect(x,y,w,h); //draw a rectangle } } } void stop() { m.stop(); //switch off camera super.stop(); } |
Version 2
Lets try out this yourself! Make use the bart.png included in source of version1.
hint: make use of what you've learn in previous lessons.
PImage, loadImage(), image()
Version 3
If you want to have a mask which is bigger / smaller than the detected face size, you can just scale up /down the image size parameter. But we have to deal with alignment problem:
To archieve that, we can breakdown this into smaller problems:
1) find out center position of a face
2) draw the image with center anchor point
Once we found the solution, we can start designing our props! Download the photoshop template:
After completing the design, hide the bottom layer and save as transparent background PNG file and put into your processing sketch. This template is based on 3*width and 3*height of the face size, so make sure we have to scale up the parameters by 3.
import pFaceDetect.*; import JMyron.*; PFaceDetect face; JMyron m; PImage img; PImage props; void setup() { size(320,240); m = new JMyron(); m.start(width,height); m.findGlobs(0); face = new PFaceDetect(this,width,height, "haarcascade_frontalface_default.xml"); frameRate(15); img = createImage(width,height,ARGB); rectMode(CORNER); noFill(); stroke(255,0,0); smooth(); props = loadImage("hair.png"); //load the props image } void draw() { background(0); m.update(); arraycopy(m.cameraImage(),img.pixels); img.updatePixels(); face.findFaces(img); noTint(); image(img,0,0); drawFace(); } void drawFace() { int [][] res = face.getFaces(); if (res.length>0) { for (int i=0;i<res.length;i++) { int x = res[i][0]; int y = res[i][1]; int w = res[i][2]; int h = res[i][3]; int x1 = x + w/2; //center X of face int y1 = y + h/2; //center Y of face int w1 = w*3; //3 * width of face int h1 = h*3; //3 * height of face pushMatrix(); translate(x1,y1); //move canvas to face position image(props,-w1/2,-h1/2,w1,h1); //render the props at center point with given size popMatrix(); rect(x,y,w,h); } } } void stop() { m.stop(); super.stop(); } |
Version 4
Here is an improvements that make use of Class concept to implement a face detector which can distinguish each face and put different features onto each face.
Algorithm
How would you do to obtain the above result? There are many ways to tackle a problem, with difference pros and cons (complexity, performance, etc...). And this is my algorithm based on their position:
At first frame, detected faces data will be assigned to create individual Candidate object. During each frame afterwards, check if there are face data near a existing Candidate, then update data of that Candidate. Otherwise (if not found) remove the candidate. In the other side, if there are new face data where no existing Candidate nearby, create a new Candidate at that position.
import pFaceDetect.*; import JMyron.*; int CAMERA_WIDTH = 320; int CAMERA_HEIGHT = 240; boolean IS_FULLSCREEN = false; FaceManager fm; //face detect + manager void setup() { if(IS_FULLSCREEN) { size(screen.width,screen.height); noCursor(); } else { size(CAMERA_WIDTH,CAMERA_HEIGHT); } frameRate(15); fm = new FaceManager(this); //initialize face manager smooth(); } void draw() { background(0); //scale up to fullscreen if(IS_FULLSCREEN) { pushMatrix(); scale((float)screen.width/CAMERA_WIDTH,(float)screen[/color ].[color=#CC0000]height/CAMERA_HEIGHT); fm.draw(); //update face manager popMatrix(); } else { fm.draw(); //update face manager } } void stop() { fm.stop(); super.stop(); } |
float DEFAULT_DAMP = 0.5; int DEFAULT_LIFE = 5; int DEFAULT_START_RENDER = 3; class Candidate { float x,y,w,h; int startcount,life; float offset; boolean[] usepic; boolean used; int id; PImage props; Candidate(float _x, float _y, float _w, float _h,int _id) { id = _id; x = _x; //center point already y = _y; //center point already w = _w; h = _h; startcount =0; life = DEFAULT_LIFE; //we call the candidate died if life = 0 (no update position in 5 frames) int ran = (int)random(3); //0,1,2 of different props if(ran == 0) { props = loadImage("hair.png"); } else if(ran == 1) { props = loadImage("hair_orange.png"); } else if(ran == 2) { props = loadImage("hair_pink.png"); } } //calculate distance between 2 points float dist2(float _x,float _y,float _w,float _h) { return dist(_x+(_w/2),_y+(_h/2),x+(w/2),y+(h/2)); } //update position (with damping) void update(float _x,float _y,float _w, float _h) { startcount++; life = 5; float len = dist(_x+(_w/2),_y+(_h/2),x+(w/2),y+(h/2)); if(len>5 ||abs( (w+h) - (_w + _h) )>7 ) { x = x*DEFAULT_DAMP + _x *(1-DEFAULT_DAMP); y = y*DEFAULT_DAMP + _y *(1-DEFAULT_DAMP); w = w*DEFAULT_DAMP + _w *(1-DEFAULT_DAMP); h = h*DEFAULT_DAMP + _h *(1-DEFAULT_DAMP); } } //lose life and render void run() { if(life>0) life--; //start render only if not dead and already appeared for continuous 3 frames if(startcount>=DEFAULT_START_RENDER && life>0) render(); } //display the candidate void render() { //scaled up by 3 float w1 = w*3; float h1 = h*3; pushMatrix(); translate(x,y); //moved to candidate position image(props,-w1/2,-h1/2,w1,h1); //draw props noFill(); stroke(0); rect(-w/2,-h/2,w,h); //draw rect popMatrix(); fill(0); textFont(createFont("arial ",20)); text("id="+id,x-w/2,y-h/2); //draw text } } |
class FaceManager { int idCount; //store id number ArrayList cans; //store all candidates PFaceDetect face; //face detector JMyron m; //camera PImage img; //camera output FaceManager(PApplet main) { idCount = 0; cans = new ArrayList(); m = new JMyron(); m.start(CAMERA_WIDTH,CAMERA_HEIGHT); m.findGlobs(0); face = new PFaceDetect(main,CAMERA_WIDTH,CAMERA_HEIGHT, "haarcascade_frontalface_alt.xml"); img = createImage(CAMERA_WIDTH,CAMERA_HEIGHT,ARGB); } void draw() { m.update(); arraycopy(m.cameraImage(),img.pixels); img.updatePixels(); face.findFaces(img); image(img,0,0); int [][] res = face.getFaces(); for(int i =0;i<res.length;i++) { float w = res[i][2]; float h = res[i][3]; float x = res[i][0] + w*0.5; float y = res[i][1] + h*0.5; float nearest = 0; int _nearest=-1; //find nearest candidate (at most of distance 50px) //if no available candidate within 50px, _nearest = -1 for(int j=0;j<cans.size();j++) { Candidate tem = (Candidate)cans.get(j); float temp = tem.dist2(x,y,w,h); if(!tem.used && tem.life>0 && temp>=0 && temp<= 50 && (_nearest == -1 || temp<nearest )) { nearest = temp; _nearest = j; } } //if found nearest candidate, update its position to new position if(_nearest>-1) { Candidate tem = (Candidate)cans.get(_nearest); tem.update(x,y,w,h); tem.used = true; } //if no near candidate, count that as a new candidate else { idCount++; cans.add(new Candidate(x,y,w,h,idCount)); } } //draw all candidates, and remove dead candidates for(int j=0;j<cans.size();j++) { Candidate tem = (Candidate)cans.get(j); tem.run(); tem.used = false; //remove if dead if(tem.life<=0) { cans.remove(j); j--; } } } void stop() { m.stop(); } } |
However, there may happen that sometimes a face cannot be detected correctly at every frame. Due to noise or lighting conditions, there are sometimes missing some faces or having more faces than expected. This is not happening very often, maybe just 1 frame every 20 or 30 frames. So that I have added startcount and life attribute to Candidate helping to check the validity of the face data. This is quite a simple and fast algorithm but there could easily be errors, such as sometimes having swaping faces or losing id on same person.
In my codes, there are two classes:
1) Candidate - where contain each candidate's attributes and functions
2) FaceManager - finding faces and analyze to assign to candidates
Changing the coes in Candidate class can change how a face behave or show, while changing FaceManager can use other methods or behaviours to judge different candidates.
Version 5
Use this as input for the mario game =]
Reference