Lesson 5 Mouse Picking

One thing every game needs is to be able to allow the player to select objects on the screen via a mouse click. 

You can continue with the lesson 1 project you've already created and modified in the subsequent lessons. For continuity and to allow you to download the lessons as separate projects I will be showing the new code in a new project called Lesson5. 

In this lesson we will learn how to set up picking by using a OpenGL pick matrix.  In other 3D APIs, like direct draw, they use ray casting through the 3D environment and test for intersection with objects in the coordinate system. OpenGL, unlike other 3D APIs, uses the rendering function to do object selection. Let me explain what that means. OpenGL takes the 3D objects in our scene and draws them onto the 2D screen we are looking at in an order depending on distance from our camera. That way the farthest objects are drawn first and the closest objects are drawn last. This allows closer object to obstruct farther objects (just like in real life). This process is called rasterization. 

When we click on the screen with the mouse the mouse sends a click event to the window with an (X, Y) coordinate location. The way OpenGL does picking is it makes an invisible matrix just like the display matrix we use to draw the screen and then instead of drawing to the screen we normally look at it draws to this "fake" screen otherwise known as the picking matrix. Each time an object is drawn to the pick matrix, if it is at the location of the mouse click,  the objects name is put on the name stack. After all the objects have been drawn we have a stack populated with the names of all the objects that were draw to the location of the mouse click. We then examine the stack and the top item is the last item pushed onto it, i.e. the closest object. We can now mark the selected object so that when it is next rendered to the visible screen it will show that is has been selected.

Lets go through what we need in order to accomplish our goals in this lesson.

1. We need a callback to capture mouse clicks.

2. We need to set up a view port and pick matrix to mimic the view port and matrix we use for normal rendering.

3. We need to modify our objects so they can render in either render mode or select mode.

4. We need to process the hits we get and mark the selected items.

Lets get started.

Click on the class tab and expand the project. Right click on MFCopenGL and choose add and add function. Add a public member function named "mouse" with a void return and 4 parameters named button, state, x, y of type int.

Add a global callback function to Lesson5Dlg.cpp just prior to the other callbacks. 

void mouse(int button, int state, int x, int y)

{

      gl.mouse(button, state , x, y);

}

In order to be able to switch between render mode and select mode we will need to move the rendering code to its own function that takes a parameter.

Click on the class tab, expand the project, right click on MFCopenGL and choose add and add function. Add a private member function named "drawShapes" with a void return and one parameter named mode of type GLenum.

We will need to store the screen width and height since we will be making more than one matrix now. Add 2 private member variables to MFCopenGL of type int named width and height. 

Change the resize function code as follows:

// The OpenGL resize callback

void MFCopenGL::resize(int w, int h)

{

      //Store the window dimensions

      width = w;

      height = h;

      //create the viewport

      glViewport(0, 0, w, h);

      //put us in projection mode

      glMatrixMode(GL_PROJECTION);

      //clear the matrix

      glLoadIdentity();

      //create the viewing frustum

      setPerspective();

}

Notice the setPerspective() function has replaced the code for setting perspective. We need add this function now.

In order to make sure our selection mode picture has exactly the same picture as our render mode picture we will put the perspective code in its own function and call it whenever we want it set. This way there is only one place in the code that ever needs to be modified if we want to change our viewing frustum. Click on the class tab, expand the project, right click on MFCopenGL and choose add and add function. Add a private member function named "setPerspective" with a void return and no parameters. I know this may seem a little strange but  not doing it this way can lead to untraceable selection bugs when the code base gets large. Here is the function code:

// Set the veiwport perspective

void MFCopenGL::setPerspective(void)

{

      //set up the frustum

      gluPerspective(45.0, (float)width / (float)height, 0.1, 1000.0); 

}

You may have noticed a few other changes in our resize function. We are going to move some of the code over into the new drawShapes function along with most of the code from our old display function. These changes are also to make sure the render and select matrixes are identical. Speaking of the display function here's our new shortened code for it:

// The OpenGL display callback

void MFCopenGL::display(void)

{    

      //set the background color to match the backColor setting

      glClearColor(backColor.r,backColor.g,backColor.b,backColor.a);   

      glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);

      glEnable(GL_DEPTH_TEST);

      drawShapes (GL_RENDER);

      glutSwapBuffers();

}

Notice the call to drawShapes? That's where all our render code went. We need to take all our drawing code and put it into drawShapes  Here's drawShapes:

// Draw shapes in render or select mode

void MFCopenGL::drawShapes(GLenum mode)

{          

      glMatrixMode(GL_MODELVIEW);

      glLoadIdentity();//clear the matrix

      // set up the viewing location 

      gluLookAt(

            0.0, 0.0, 200.0, // eye location

            0.0, 0.0, 0.0, // center location

            0.0, 1.0, 0.0); // up vector

      //iterate through the cube array, animate and draw them

      for(int x = 0; x < 4; x++)

      {

            cube[x].tick();

            cube[x].draw(mode);

      } 

      for(int x = 0; x < glObjectArray.GetCount() ; x++)

      {

            glObjectArray[x].tick();

            glObjectArray[x].draw(mode);

      }

      glutPostRedisplay(); //repaint the display   

}

Notice the cube[x]draw(mode); call. We will pass the drawing mode on to the object so it knows what drawing mode we are in. Next we need to add code to the glObject to detect when we are in select mode and if we are in select mode push some name ids onto the name stack. Click on the class tab, expand the project, and expand the glObject class. Double click on the draw function and modify it so that it takes a parameter named mode of type GLenum. Now double click on the glObject class name and modify the entry in the header for the draw function to reflect the change we made to the parameter list.

Header:

Each object in our scene will need a unique id number and bool to track whether it is selected or not. Add a public GLuint named id and a public bool named selected to glObject class.

Modify the MFCopenGL default constructor to give each of the cubes in the array a unique id. Add the following to the top of the constructor:

      //set the cube ids to unique ints

      for(int x = 0; x < 4; x++)

      {

            cube[x].id = x;        

We also need to store the size of the pick square. The larger the pick square the sloppier the user can be with the mouse and still get a hit, but if you get the square to large it can be difficult to select small objects that are close together. Add 2 private int variables named pickWidth and pickHeight. In the default constructor set them both equal to 2. This will give us a 2 x 2 cursor hot spot to use for selection.

The whole constructor:

Modify the draw function to check for select mode:

// Draw the object in the scene
void glObject::draw(GLenum mode)
{
glPushMatrix();// save matrix prior to transform
glScalef(scale.x,scale.y,scale.z); //scale the cube on all 3 axis
glTranslatef(loc.x, loc.y, loc.z);
glRotatef( rotation.x, 1.0, 0.0, 0.0 );//rotate about the X axis
glRotatef( rotation.y, 0.0, 1.0, 0.0 );//rotate about the Y axis
glRotatef( rotation.z, 0.0, 0.0, 1.0 );//rotate about the Z axis
if(selected)
{
glColor3f(1 , 0.0, 0.0);
glutWireSphere(15, 10, 10);
}
if (mode == GL_SELECT)
{
// put names on name stack
glPushName(id); // 0 - numOfObjects
}
// 8 unique points in a cube
float vert_xyz[8][3] =
{
{ -10.0, -10.0, -10.0 }, // point 0
{ -10.0, -10.0, 10.0 }, // point 1
{ -10.0, 10.0, -10.0 }, // point 2
{ -10.0, 10.0, 10.0 }, // point 3
{ 10.0, -10.0, -10.0 }, // point 4
{ 10.0, -10.0, 10.0 }, // point 5
{ 10.0, 10.0, -10.0 }, // point 6
{ 10.0, 10.0, 10.0 } // point 7
};
int tri_abc[12][3] =
{
{0,2,4}, {4,2,6}, // Back
{0,4,1}, {1,4,5}, // Bottom
{0,1,2}, {2,1,3}, // Left
{4,6,5}, {5,6,7}, // Right
{2,3,6}, {6,3,7}, // Top
{1,5,3}, {3,5,7} // Front
};
//the colors of each of the triangles
float colors_rgb[12][3] =
{
{0.5f,0.1f,0.1f }, {1.0f,0.1f,0.1f }, // Red
{0.5f,0.5f,0.1f }, {1.0f,1.0f,0.1f }, // Yellow
{0.1f,0.5f,0.1f }, {0.1f,1.0f,0.1f }, // Green
{0.1f,0.5f,0.5f }, {0.1f,1.0f,1.0f }, // Cyan
{0.1f,0.1f,0.5f }, {0.1f,0.1f,1.0f }, // Blue
{0.5f,0.1f,0.5f }, {1.0f,0.1f,1.0f } // Magenta
};
// heres where we iterate through the triangle list
int iTriTotal = 12;
int iTriIndex = 0;
glBegin(GL_TRIANGLES); // this tells OpenGL to change the state to drawing triangles
for ( iTriIndex = 0; iTriIndex < iTriTotal; iTriIndex++ )
{
glColor3fv( colors_rgb[iTriIndex] );//set the color of the vertex
glVertex3fv( vert_xyz[tri_abc[iTriIndex][0]] );//set the A vertex of the triangle
glColor3fv( colors_rgb[iTriIndex] );//set the color of the vertex
glVertex3fv( vert_xyz[tri_abc[iTriIndex][1]] );//set the B vertex of the triangle
glColor3fv( colors_rgb[iTriIndex] );//set the color of the vertex
glVertex3fv( vert_xyz[tri_abc[iTriIndex][2]] );//set the C vertex of the triangle
}
glEnd();// take OpenGL out of the draw list state
if(mode == GL_SELECT)
{
// remove names from name stack
glPopName(); // pop the value off
}
glPopMatrix();// restore matrix after transform
}

Indented:

Notice that the check for selected mode occurs both before and after the rendering code. All drawing must be preceded by a glPushName and followed by a glPopName. That way the OpenGL can associate the object rasterized to the screen with its id number. Remember never draw anything to the screen during selection mode unless it is bracketed by push and pop in this way. If you have objects that you don't want the player to be able to select then detect selection mode and don't draw that object when in selection mode.

Now that we have code in place to handle rendering in selection mode we need to call it when the user clicks the left mouse button on one of the cubes. Click on the class tab, expand the project, expand the MFCopenGL class and double click on the "mouse" function. Here is the mouse event code:

The whole function with indents:

Notice the call to setPerspective. Now we can be sure that the pick matrix has the same frustum as the render matrix. Notice the processHits function, we need to add that now. Click on the class tab, expand the project and right click on the MFCopenGL class and choose add and add function. Add a private member function named processHits. You will need to manually modify the header and the function in order to pass an array as a parameter. The wizard interface doesn't like to see [] in a name. The header line should look like this:

void processHits(GLint hits, GLuint buffer[]);

The function line should look like this:

void MFCopenGL::processHits(GLint hits, GLuint buffer[])

Here's the code for processing the hits:

int y = 0;
//unselect all
for(int j = 0; j < 4; j++)
{
cube[j].selected = false;
}
if(hits > 0)//Make sure there is at least one hit
{
//The 4th spot, array index 3, is the top of the stack,
//which holds the id of the last item that was drawn to the place we clicked on
//The last item would be the one closest to us since items are drawn
//farthest first to closest last
y = buffer[3];
if((y>=0)&&(y<4))//sanity check, ids 0-3
{
cube[y].selected = true;//mark the selected cube
}
}

The whole function with indents:

Compile and run and hit play. You should now be able to click on one of the cubes and it should be encased in a frame sphere when selected. Notice how the 3rd cube is selected in the following screen capture.

Setting up picking can be tricky. Never leave it to add later in a project. Always add picking when the project is still small and manageable. That way if as the project grows and the selection stops working you will know the problem is in the last code added. 

You can download the source code for this lesson here.