Previously
This is the fourth in a series about TDD’s value to humans, following
- Keeping my job, in which I lost my manager’s trust,
- A last-minute feature, in which I regained some, and
- Deadline pressure, in which I earned quite a bit more than I’d started with
Same idea, next level up
As a developer, I’d significantly reduced the waste and risk in my work by incrementally bringing our legacy code under test while test-driving new features (and bug fixes). As a newly minted product manager with a small budget and high customer expectations, if I wanted to maximize my tiny team’s business impact, I needed to find our next big waste-and-risk targets and bring them down.
Conveniently, they were the same target. We’d been releasing into production roughly every 3-6 months. That meant waste: customers had to wait a long time to get any value from our work. And it meant risk: we had to worry about a whole lot of haven’t-thought-about-those-in-a-while changes going into production all at once.
Doctor, it hurts when I do this
How had we settled on this release (in)frequency? Because originally, deciding when to release hadn’t been easy. On our end, having had no immediate feedback about the state of our code, we lacked ongoing confidence in it, so we had to steel ourselves and others for any change, so we released only when someone wanted the desired value badly enough. (Thereby perpetuating the cycle? Naturally.) On Operations’ end, supporting many products, their job was to review all proposed changes to production and reject unsafe ones, and they’d learned to treat even safe-looking changes as unsafe until proven otherwise. They’d set the bar high, with good reason.
If it hurts, do it more often
But by giving ourselves an immediate feedback mechanism, our part of the decision had become easy. As far as we were concerned, the meaningful and thorough automated tests we’d built put us well over that bar. At each desired release, our thousands of green tests (and zero red ones) covered not only the new functionality but also everything that had ever been tested before. Which, by now, was rather a lot. Of course we were confident to ship. Therefore, in the interest of minimizing branch-management effort, lead time, and release-process forgetfulness, we asked Operations for special permission not to have to freeze our code a week in advance, but to ship everything that had been done up to and including release day. Releases were on Friday evenings, and Ops needed a few hours to prep what we gave them, so that meant we had till early afternoon to get in whatever we could get in. (Given the time constraint, not a good day to do something complex or risky, but perfectly fine to try to squeeze in something simple and valuable.)
Operations saw that we were confident, and they saw our test results, and they were tentatively approving our non-standard releases — until one release day, after a teammate had pulled an all-nighter to get one last piece in, when Ops decided at the last minute that the release was no longer acceptable. Not because of anything my teammate had or hadn’t done, but because I hadn’t added a couple lines of documentation about something minor to the Ops wiki in advance of the release procedure. I’d been planning to make those tiny edits during the several Friday-evening hours I’d be on the phone with Ops alternately guiding them through the release and idle-waiting for the next step in our procedure.
If it hurts a lot, maybe take a step back
There would’ve been plenty of time. But they chose to cancel the release. As was their right. Mine, in turn, was to react by huffing and puffing about the spurious reasoning for the last-minute change of plan and the evident disrespect for my teammate’s effort. Predictable lot of good that did!
When I’d gotten that out of my system at everyone else’s expense, I had the whole weekend to think quietly about my actions before, during, and after, and about why Operations might have felt the need to choose as they did. They did not, by default, trust engineering teams to deliver with safety. They could not. It was their job to be skeptical, and on the rare occasions they hadn’t been sufficiently so, they’d been hurt. And here I was asking them to trust us not only to deliver with safety, but to let us skip a very visible part of the review process. I was asking them to let us deliver whatever we — not they — decided was ready, right up to the last minute, but I hadn’t shown them why they ought to, certainly not in terms that made sense to them. So they were likely inclined to feel that I was asking them to be negligent in their roles, and they were likely inclined to perceive any small failing of documentation as a red flag indicating sloppiness elsewhere. Now that I saw where they might be coming from, I couldn’t blame them. I wouldn’t have wanted to be negligent either.
Working together for shared understanding
Because I had retrospected, by the time we got together to discuss what had happened, I knew what I wanted to try. I apologized for my outburst and asked Luke, the Operations manager, if he’d be willing to pair with me to develop a new feature sometime. To my relief, and his great credit, he agreed.
It wasn’t long before we had our chance. Not unlike in A last-minute feature, a small feature request came in on a Wednesday, hoping to be delivered as part of the release scheduled for that Friday. For any other Engineering team, the rule about freezing a week in advance would have made this a non-starter. But this was exactly the kind of (apparently somewhat common) situation we wanted to handle with agility. And it was a small, simple set of new behaviors, perfect for pairing with someone who didn’t know our code but knew our system very well.
Luke and I got together Thursday in a conference room. I had set him up with access to our source code repositories and we started by running the tests. We broke something obvious and saw that some tests went red. Then we reverted to green and talked through what the new feature would look like when it was working right. When we had a shared understanding, I talked him through writing the equivalent test assertions. He ran them; red as expected. I talked him through how to change the code. He ran the new tests again; green as expected.
”That’s the whole feature. Cool, right?” I asked.
”Yeah, sure, kinda,” Luke allowed.
An unexpected twist
”Okay, before you check it in, let’s run the whole test suite.”
Whoa. Some other tests, seemingly unrelated, were now red. Hadn’t seen that coming. A smile came over Luke’s face. ”Now I get it,” he said. “This problem can never happen in production. We found the problem before we ever checked it in.” Exactly! Same for everything else we’ve got tests for. (And, boy howdy, have we got a lot of tests.)
After quickly fixing the fallout, Luke committed his new feature. It went into production on the following evening.
Conclusion
For as long as Luke was in that role, we had no further difficulty delivering just-in-time releases. And because we knew the sailing would be smooth, we began releasing once a month, wasting less and risking less — just as I’d hoped. More often was too often (busy work, lost Friday nights); less often wasn’t often enough (Eng and Ops both out of practice at release procedure); once a month was just right.
While continuous delivery was prohibitively difficult in our tightly controlled environment, we managed to achieve continual delivery on a cadence that met everyone’s needs. For Operations, we controlled the inherent risk of change by doing it in smaller batches, giving us less to roll back and binary-search through in case of trouble. For customers, we could honestly offer the possibility of helping them as soon as next month, limited only by our own constraints (priorities, WIP, nature of the request). And for ourselves, since we’d always be delivering to production soon, we could see more clearly the value and relevance of our work, and hope for more meaningful customer feedback about it.
TDD gave me an opening. Luke’s willingness and ability to collaborate (and to forgive my outburst) allowed us to walk through it together, making a sizable impact on the business we both served — and on me.