An Embarrassing Bug
Posted by Charlie Thu, 14 Feb 2008 06:44:00 GMT
<p>Its always embarrassing when critical bugs slip through our testing and get deployed to production. Over the weekend we deployed an update the broke our reset password functionality. Thus if you didn't remember your password, then there was no way to get it back. Luckily a couple of users quickly alerted us to the problem.</p>
<p>The reset password functionality is some of the oldest code in the system and has full functional test coverage, but not selenium test coverage. The difference is that functional tests are run from a command line while selenium tests are run from a browser.</p>
<p>So what happened? Of course all of the functional tests passed, otherwise the deployment would have failed. What happened was the tests were wrong. A few weeks ago we made a couple of changes to better protect passwords to prevent them being logged to our bug database. Those changes caused the reset password interface to change slightly, but the old one was left in place by mistake. So the functional tests were exercising the older interface, while clients started using the newer untested interface. And thus the problem slipped through.</p>
<p>Looking back, we made two obvious mistakes. First, the older interface should have been immediately removed. Second, we did not have adequate Selenium test coverage. Both issues are now fixed, so I feel pretty confident that bug won't create back into the system again.</p>