So, the first step in implementing this feature is to create the ray to intersect with the scene. The simplest way to do this in my engine is to invert the 'ViewProjection' matrix used in normal rendering. Then, you use the mouse position within the window to calculate it's clip space position on the far clipping plane. For example, if the mouse was right in the middle of the window, the clip space position at the far plane would be [0.0, 0.0, 1.0, 1.0] (this is assuming the LH coordinate system from D3D - OpenGL simply points in the other direction along the z-axis). Taking this point and transforming it by the inverse of the 'ViewProjection' matrix give the world space position on the far clipping plane.
Next, we need the viewer's position. This can be found with another matrix inversion - this time only the 'View' matrix needs to be inverted. If you transform the point [0.0, 0.0, 0.0, 1.0] with the inverse of the view matrix, then it gives you the world space position of the camera (by virtue of the definition of view space, the origin is the viewer's position!).
With both of these points, I can construct the ray that I need. However, instead of simply using some black box function, I decided to read up on the math behind matrix inversion and see how it is actually calculated. How many of you out there can say exactly how you invert a matrix? I know I couldn't, so I wanted to figure it out. I searched around to the usual suspects like mathworld.wolfram.com and a few others, and it seemed that there weren't any consistant descriptions of how to invert a matrix. Admittedly, I may have been looking in the wrong places, but it seemed very confusing and contradictory - each equation worked a little different. I began to suspect that there was a more generalized set of equations for finding the inverse that the various sources were making assumptions about. Clearly, the technique required using Laplacian Expansion, but there were only fuzzy descriptions of the technique and I wanted to be more clear about it before spending time to implement a technique that I fully understand yet.
Then, I remembered why I so often refer to the Dave Eberly books and his website, geometrictools.com. After about two minutes of looking, I found this paper on Laplacian Expansion and Inverting Matrices. This paper describes taking determinants of matrices by using Laplacian Expansion on 2x2, 3x3, and 4x4 matrices. The paper is detailed enough to understand, but not so technical that it takes a week to decipher either. To make things even better, he shows how to apply Laplacian Expansion to 4x4 matrix inversion in a non-obvious way that saves a significant amount of work.
I don't know what drives Dave, but his information is always accurate and succint. In addition, he responds to email questions on his books, publishes all of his engine source code for free, and writes great little articles like the one above to help out. If you don't have one of his books yet, go order one - I have used it many, many times and haven't been let down yet!
Now I can implement my picking ray and understand what I'm doing at the same time!!!
