I spent several hours today (well, most of it was yesterday) tracking down a bug in some experimental vector math code of mine. It's the kind of stuff that's sensitive to real-time behavior, so you can't step through it in a debugger without ruining the numbers - scrubbing through a few MB of logs is the only way to find out what's going on.
For instance, at one point during this process, I had a couple of angle variables and was freely copying values between them. Thanks to some poor naming on my part, I was feeding radians into a segment of the calculation that expected degrees. (I later discovered that, thanks to someone else's poor namings, the part that was in degrees was wrong, and the whole darn thing should have been in radians to begin with.)
It was downhill from there.
I'll spare the long tale of woe and boredom (actually, it wasn't that bad - most of the time was spent 30 miles away at a poker game) and get to the punch line. I had two vectors, vref and vcam. They were to be adjusted via a series of rotations, producing the results av and avcam.
My final output was relying on the raw vectors, vref and vcam. The net result was bogus data getting fed down the pipeline; the visible result of that was chaotic behavior that was nearly impossible to trace to specific stimuli. In other words, the most pernicious type of bug. The type that makes a 30 minute prototyping task chew up 5 hours.
Whoops.
