I've written of my love for Test-Driven Development and for ikiwiki. Love is awfully abstract, though. Here's a concrete example.
Some context
Ikiwiki is a wiki compiler. It transforms easy-to-write input files into easy-to-browse output files.
Since a wiki is open for editing, it's important to be able to see what changed recently. Typical wikis provide their own limited way to view and compare a limited number of revisions. Ikiwiki integrates with many popular revision control systems.
The integration is deep. Write in the browser, and saving an edit means making a commit. Write in the terminal, and committing to the repository means regenerating output files.
I developed and contributed ikiwiki's CVS integration a few years ago. It's been in production use ever since.
The bug
On cvs commit, the commit succeeded and affected output was regenerated, but with a warning:
Use of chdir('') or chdir(undef) as chdir() is deprecated at /usr/pkg/lib/perl5/vendor_perl/5.16.0/File/chdir.pm line 45.How it got through
The warning was introduced in Perl 5.12. I had written the code under Perl 5.10. (The warning appeared harmless, and I was busy, so Perl got all the way to 5.16 before I could deal with it.)
How I came to understand it
- Reproduced the problem by hand.
- Extended my tests to automatically reproduce the problem.
- Tried to reach understanding through thinking.
- Succeeded at reaching understanding through debug
printf()s.
How I fixed it
- Tried some things that made the tests look good.
- Kept the simplest one.
How it'll stay fixed
If this were a canonical example of bugfixing with TDD, I would have written a test that fails if the warning appears and succeeds otherwise. Once the bug was fixed, that test would have magically become a regression test.
This isn't a canonical example of bugfixing with TDD. The bug manifested with ikiwiki as a post-commit hook, which my tests hadn't been covering. It took me a while to figure out how to wire up the needed integration in test. So when I started seeing the warning as a side effect of other tests, that was good enough for me at the time.
Writing these words spurred me to revalidate my previous judgment. I just tried reverting my fix and running the tests. As long as I don't run them verbosely — and I usually don't — the warning that reappears is very hard to miss. Still, I didn't write, and now don't have handy, a purpose-specific test that unambiguously fails if stderr matches the warning. That would be an obvious boon to future developers who don't know what I currently know (a set that includes me) or who don't always run the tests the way I mostly do (also a set that includes me), as well as a boon to the didactic value of this real-world example.
Pictured
You can see that I spent some time:
- Improving automated tests
- Fixing a bug
Not pictured
No code artifact can show you how much time I saved by:
- Making the computer tell me instantly whether I've solved the problem yet
- Not needing to settle for a less elegant fix
- Almost certainly never seeing this bug again
Significance
As I've learned to expect, the up-front cost of making tests help me was nowhere near the benefit, especially over the long lifetime of production code. I'm pretty sure I got done faster (and better). If you write code for a living and you don't believe me, you owe it to yourself to question the basis of your belief.