At Agile Testing Days, I facilitated a workshop called “DevOps Dojo”. We role-played Dev and Ops developing and operating a production system, then figured out how to do it better together.

Some firsts


I’ve spoken at several instances of pkgsrcCon (including twice in nearby Berlin), but that’s more like a hackathon with some talks. Agile Testing Days was a proper conference, with hundreds of people and plenty of conferring. If someone asks whether I’m an “international speaker”, or claims I am one, I now won’t feel terribly uncomfortable going along with it.


Two WeDoTDDers

I met a fellow WeDoTDD practitioner, Carlos Blé of Codesai. (Here’s their WeDoTDD interview and Pillar’s.) Carlos and I have both relied on Twitter to build our careers. Who knows, maybe we’ll give a talk together about it.


At the Tuesday morning Lean Coffee, I found a bug in myself (not a first).

What I expected from many previous Lean Coffees: I’d have to control myself to not say all the ideas and suggestions that come to mind.

What happened at this Lean Coffee: It was very easy to listen, because I didn’t have many ideas or suggestions, because the topics came from people who were mostly testers.

Conclusions I immediately drew:

  • Come to think of it, I have not played every role on a team. I don’t know what it’s like to be a tester. Maybe my guesses about what it’s like are less wrong than some others, but they’re still gonna be wrong.
  • This is evidently my first conference that’s more testing than Agile. Cool! I bet I can learn a lot here.


Four Yorkshiremen

Matt Heusser invited me to performed in the Late Night Cabaret with Bob (Lisa Crispin’s spouse) and Jack (Janet Gregory’s). We did the Four Yorkshiremen sketch as popularized by Monty Python.


Thanks to Troy Magennis, Markus Gärtner, and Cat Swetel, I decided to try a new idea and spend a few slides drawing attention to the existence and purpose of Agile Testing Days’ Code of Conduct. I can’t tell yet how much good this did, but it took so little time that I’ll keep trying it in future conference presentations and workshops.

Some nexts


My next gig will be remote coaching, centered around what we notice as we’re pair programming and delivering working software. I’ve done plenty of coaching and plenty of remote work, but not usually at the same time. Thanks to Lean Coffee with folks like Janet and Alex Schladebeck, I got some good advice on being a more effective influencer when it takes more intention and effort to have face-to-face interactions.

  • Alex: For a personal connection, start meetings by unloading your “baggage” — whatever’s on your mind today that might be dividing your attention — and inviting others to unload theirs. (Ideally, establish this practice in person first.)
  • Janet: Ask questions that help people recognize their own situation. (Helping people orient themselves in their problem spaces is one of my go-to strengths. I’m ready to be leaning harder on it.)


As I learn about remote coaching, I expect to write things down at Shape My Work, a wiki about distributed Agile that Alex Harms and I created. You’ll notice it has a Code of Conduct. If it makes good sense to you, we’d love to learn what you’ve learned as a remote Agilist.


I found Agile Testing Days to be a lovingly organized and carefully tuned mix of coffee breaks, efficiency, flexibility, and whimsy. The love and whimsy shone through. I’m honored to have been part of it, and I sure as heck hope to be back next year.

We’d be back next year anyway; we visit family in Germany every December. Someday we might choose to live near them for a while. It occurs to me that having participated in Agile Testing Days might well have been an early investment in that option, and the thought pleases me. (As does the thought of hopping on a train to participate again.)


I’m in Europe through Christmas. I consult, coach, and train. Do you know of anyone who could use a day or three of my services?


One aspect of being a tester I do identify with is being frequently challenged to explain their discipline or justify their decisions to people who don’t know what the work is like (and might not recognize the impact of their not knowing). In that regard, I wonder how helpful Agile in 3 Minutes is for testers.

Let’s say I could be so lucky as to have a few guest episodes about testing. Who would be the first few people you’d want to hear from? Who has a way with words and ideas, knows the work, and can speak to it — in their unique voice — to help the rest of us understand a bit better?

Posted Wed Dec 7 04:01:44 2016 Tags:

My first job was in Operations. When I got to be a Developer, I promised myself I’d remember how to be good to Ops. I’ve sometimes succeeded. And when I’ve been effective, it’s been in part due to my firsthand knowledge of both roles.

DevOps is two things (hint: they’re not “Dev” and “Ops”)

Part of what people mean when they say DevOps is automation. Once a system or service is in operation, it becomes more important to engineer its tendencies toward staying in operation. Applying disciplines from software development can help.

These words are brought to you by a Unix server I operate. I rely on it to serve this website, those of a few friends, and a tiny podcast of some repute. Oh yeah, and my email. It has become rather important to me that these services tend to stay operational. One way I improve my chances is to simplify what’s already there.

If it hurts, do it more often…

Another way is to update my installed third-party software once a week. This introduces two pleasant tendencies: it’s much…

  1. Less likely, at any given time, that I’m running something dangerously outdated
  2. More likely, when an urgent fix is needed, that I’ll have my wits about me to do it right

Updating software every week also makes two strong assumptions about safety (see Modern Agile’s “Make Safety a Prerequisite”): that I can quickly and easily…

  1. Roll back to the previous versions
  2. Build and install new versions

Since I’ve been leaning hard on these assumptions, I’ve invested in making them more true.

The initial investment was to figure out how to configure pkgsrc to build a complete set of binary packages that could be installed at the same time as another complete set. My hypothesis was that then, with predictable and few side effects, I could select the “active” software set by moving a symbolic link.

It worked. On my PowerPC Mac mini, the best-case upgrade scenario went from half an hour’s downtime (bring down services, uninstall old packages, install new packages, bring up services) to less than a minute (install new packages, bring down services, move symlink, bring up services, delete old packages after a while). The worst case went from /!\ over an hour to {OK} maybe a couple of minutes.

…Until it hurts enough less

I liked the payoff on that investment a lot. I’ve been adding incremental enhancements ever since. I used to do builds directly on the server: in place for low-risk leaf packages, as a separate full batch otherwise. It was straightforward to do, and I was happy to accept an occasional reduction in responsiveness in exchange for the results.

After the Mac mini died, I moved to a hosted Virtual Private Server that was much easier to mimic. So I took the job offline to a local VirtualBox running the same release and architecture of NetBSD (32-bit i386 to begin with, 64-bit amd64 now, both under Xen). The local job ran faster by some hours (I forget how many), during which the server continued devoting all its I/O and CPU bandwidth to its full-time responsibilities.

Last time I went and improved something was to fully automate the building and uploading, leaving myself a documented sequence of manual installation steps. Yesterday I extended that shell script to generate another shell script that’s uploaded along with the packages. When the upload’s done, there’s one manual step: run the install script.

If you can read these words, it works.

DevOps is still two things

Applying Dev concepts to the Ops domain is one aspect. When I’m acting alone as both Dev and Ops, as in the above example, I’ve demonstrated only that one aspect.

The other, bigger half is collaboration across disciplines and roles. I find it takes some not-tremendously-useful effort to distinguish this aspect of DevOps from BDD — or from anything else that looks like healthy cross-functional teamwork. It’s the healthy cross-functional teamwork I’m after. There are lots of places to start having more of that.

If your team’s context suggests to you that DevOps would be a fine place to start, go after it! Find ways for Dev and Ops to be learning together and delivering together. That’s the whole deal.

Here’s another deal

Two weeks from today, at Agile Testing Days in Potsdam, Germany, I’m running a hands-on DevOps collaboration workshop. Can you join us? It’s not too late, and you can save 10% off the price of the conference ticket. Just provide my discount code when you register. I’d love to see you there.

Posted Wed Nov 23 19:35:52 2016 Tags:

Every four years, I revisit my thinking about voting. I do it because I’ll revisit any old thinking given a chance and a reason, and because one of the consequences of this particular old thinking is that many people, on hearing it, feel more distant from me.

A theme of my life has been to find steps I can take to feel less apart from other people. Every election has generally been a big backward step. Every four years, I hope I can adapt myself to holding less alienating thoughts, or at least find less alienating ways to express them.

Right now, I’m feeling hopeful.

I believe, and you may agree, that…

It was meaningful and emotionally resonant when we managed to elect a black man as President. It will be meaningful and emotionally resonant when we manage to elect a woman. Maybe today.

I’m open to the possibility that, in some contexts…

  • Voting is morally or ethically superior to not voting
  • Voting can more effectively bring about desirable outcomes than not voting
  • Voting is significant compared to other actions we can take

I have not yet found myself in contexts where these claims seemed likely to be true.

I agree that…

Voting is buying into democracy.
Jabe Bloom

In previous elections, I’ve often been told “If you don’t vote, you can’t complain” (about what the elected officials wind up doing). I’ve sometimes replied “If you do vote, you can’t complain” (about what the system of power inevitably winds up doing). But I never found it satisfying to say that. Everyone can always complain whenever they like, of course, and my goal had been to increase mutual understanding, not score a cheap rhetorical point.

In this election, I don’t recall anyone telling me this one. I’d be happier about it if I didn’t think it was because I’ve been talking openly with many fewer people than in previous elections. (I’m writing these words in a small attempt to do better about that.)

In previous elections, I’ve often been told that the debates are an important way to become informed about the issues and the candidates, and it’s irresponsible not to watch. In this election, I haven’t heard anyone say this. I’m somewhat happy about it because it means, for once, that my not watching the debates hasn’t added distance between me and other people. I’d be happier about it if we weren’t so polarized that those who watched the debates saw only confirming evidence of their choice.

I don’t agree that…

not voting is self-disenfranchisement
Jabe Bloom

I’m not disenfranchised simply because I don’t want to be part of this specific franchise. I continue to exercise my agency and influence in ways I find congruent.

It matters who we elect

It also costs us. Some of the costs we never know, and many we can only guess. By the time we know more, it’s too late.

We all suspect we know which choice of presidential candidate would reduce more harm. One of them sounds, to my ears, far more evidently dangerous. That doesn’t fill me with certainty. It fills me with doubt about the other candidate, who is far more practiced at appearing sensible and humane and also at working the system of power.

I know which one sounds worse. But I don’t know which one is worse. We can’t know. There is no means through which that knowledge can be available to us.

Do I want to participate in a giant system of power — one with with long, slow, obfuscatory feedback cycles — that requires us to put one of these people in office? I don’t. As an Agilist, I don’t see this as a system that can ever work well, or that can ever trend toward better.

There’s no way this election doesn’t have consequences. That doesn’t mean I see any option for influencing them to my liking.

It also matters, far more, what else we do

John Lewis risked his life for something he held dear:

I’ve marched, protested, been beaten and arrested — all for the right to vote. Friends of mine gave their lives. Honor their sacrifice. Vote.
John Lewis

Selma flattened me. I don’t know that I could have done what he did. I do know that the value he places on voting does not obligate me to do the same. When I see how he and others had to fight for some small additional fraction of equal treatment backed by the power of law, it does not occur to me that to solve this problem we must arrange for this power to be shared by more people. It occurs to me that this power is the proximate cause of the problem. We can’t prevent cruelty, but we are not obligated to put leaders in a system of power that magnifies the effects of any cruel behavior. And if the only option we can offer those to whom this system of power has been especially cruel is for them to participate in it more fully, then there is much more kindness waiting to be released from our hearts.

We invented this system. We can invent another one.

Neither voting nor not voting is sufficient

That’s how I see it. If voting is part of what you need to do, I’ll try to understand. I hope you’ll try to understand why not voting is part of what I need to do. And I’m quite sure that together, with our combined privilege and savvy and smarts and love, we can have a far more meaningful influence than the one we’ll have today.

Let’s keep using all our advantages to move the world, inch by inch, toward the world we want.

Posted Tue Nov 8 08:02:26 2016 Tags:

There was an Agile Coach Camp two years ago in Indianapolis — I could have been making crucial career-changing connections, then sleeping in my own bed! — but we were out of the country that weekend. It was, unquestionably, a missed opportunity; perhaps what happened was the only thing that could have. Things worked out anyhow. Pillar gave me the work, and I’ve been trying to make the most of it ever since.

I’m mildly amused at the timing of my first Agile Coach Camp. Before I started down the coaching path, I imagined that sustaining it — and myself — would mean I’d need to alternate between “doing it” and “helping others do it”, like it says right there in the Manifesto. So far, I’m not wrong. At my request, I’ve just swung my professional pendulum most of the way back to “doing it”, recharging my coaching energy and developer cred on a team of fellow consulting-minded XP developers. We’re embedded at a client, working to influence the outcome of a change-the-world project that’s in the news. (Interested? Hit me up.)

Maybe this mental context switch is why, when only a handful of Campy Coaches self-identified as “technical”, I was primed to figure out how I might be able to share with the rest of them a tiny experience of software development work. En route to St. Louis, I’d already been revisiting this old thought:

People who haven’t done it have no way of knowing the extent to which programming consists of making decisions. (It’s a very large extent.)

The first time I wrote these words down was when I resigned from Morgan Stanley. Three years ago, despairing at how few around me shared my values, I discovered the Software Craftsmanship community. (If they sound like your people too, Global Day of Code Retreat is this Saturday, and I highly recommend participating.) I’ve had cause lately to reflect on how much my life has improved, including how my role in the community has changed; if I hadn’t, spending a long weekend with Agile friends in the venue’s main meeting room surely would have started the gears turning.

the Morgan Stanley room

I came up with a few ways to offer folks an introduction to important stuff that might normally make them uncomfortable:

  1. Mob Programming ESPECIALLY for Non-Programmers, during which we used Vim to write C and almost got one FizzBuzz test to pass
  2. Test-Driven Development and Pairing for Non-Techies with Mark Balbes, during which we “test-drove” and “refactored” a bit of prose
  3. ”Show Me The Tech”, in which anyone interested to try/see/discuss any technical practice could pull me aside anytime for 15 minutes

The feedback I received suggested that this is a direction I might want to keep going:

Closing circle at #ACCUS. My key learning: I can do a lot to help non-developers better understand the work, and it’s needed and wanted.

Thank you to Jason Tice for giving us this Open Space, and to everyone who joined me in finding meaningful ways to fill it.

Posted Sun Oct 16 20:12:37 2016 Tags:

I’ve been running a public web server since 1999, when my employer registered for me as a gag gift. Last week, I learned from Twitterbrausen that in German, “Schmonz” means something akin to “bullshit”. That’s not what my employer had meant by it; I consider nonetheless that my incessant blogging has acquired a fine new patina of significance.

As I recall, when I was first looking for web server software, there was not a wide variety to choose from. Apache was popular and featureful, a safe default choice. As a novice programmer, I was very much taken with the idea of building dynamic sites, and Apache offered many ways to go about that. Done deal.

In the intervening years, my server machine has changed several times, from Macintosh IIci to Mini-ITX box to Mac Mini to Xen Virtual private server. (I’m particularly fond of the present arrangement wherein hardware is someone else’s problem and I continue to have root access.) No matter the system architecture, the OS has always been NetBSD, which remains unobtrusively thrilling, and the web server has always been Apache, which has gradually become more noisome.

Between my own sites and those of friends I’ve hosted, I’ve needed many times to adapt my Apache configuration to accommodate changes in external modules (such as mod_php), to interfaces (such as PHP via FastCGI instead), and within Apache itself (such as basic access control). Each time I forcibly revisited my config, I found myself revisiting my discomfort with its complexity. I never felt sure that I understood exactly, in its entirety, what my Apache installation would and wouldn’t do. And as a result of years of entanglement and unclarity, I never saw a way to give my users full administrative control over their own sites.

I’ve been imagining moving off Apache for a while. But it always seemed like a project, so I never did anything about it. I can’t usually afford to start on something unless I know I’m going to be able to stop soon, and I won’t usually want to stop unless I know how I can easily start next time. That leaves me needing a sequence of small-enough steps in my desired direction. Or, more precisely, two expectations: that at least one such sequence exists, and that I’ll be able to discover one as I go.

Conveniently, I’ve had plenty of professional practice at incremental problem-solving, enough to identify my first few steps and start making progress. Here’s the rest of the sequence, naming the refactorings I’ve found along the way.

Step 1: Extract Virtual Host

I wanted to see what I’d learn by persuading one site to become its own self-contained thing running its own Apache instance. I picked a relatively basic site, told the system Apache to reverse-proxy that virtual host, added just enough configuration to start a site-specific Apache on localhost, verified that as far as I could discern the site worked equally well, and cut over to the new configuration.

Inserting a proxy usually means, at the very least, server logs start reporting requests coming from the proxy’s IP rather than the browser’s. For this to be a refactoring, the system Apache needed to send an X-Forwarded-For header (it automatically does), and the site-specific Apache needed to know to look for it (by enabling the bundled mod_remoteip).

Manually starting an instance of a service usually means the system won’t automatically know how to do the same next time it boots up. For this to be a refactoring, I needed to add an entry to the site owner’s crontab. To validate that the site would continue to be served by its own Apache as well as it’d been served the old way, I rebooted the system. The site stayed up.

Step 2: Extract More Virtual Hosts

Good, because there were 17 more sites to go. Each of them would also be listening on its own non-standard port on localhost. To identify them at a glance in netstat, I added the port to /etc/services. Now I had a pattern worth repeating.

Some sites were more complex than others (PHP, language negotiation, other wrinkles), but I didn’t need to invent their configurations from scratch, merely uncover the tiny portions of the existing giant config that were relevant and copy them over.

Near the end, I couldn’t start new Apache instances without increasing some kernel IPC parameters (kern.ipc.msgmni from 40 to 80, kern.ipc.semmni from 10 to 20). This felt like a small backward step. I hoped to be able to undo it later.

It also might have felt like a small step backward to suddenly have lots more instances of Apache. But it was a large step forward in my understanding.

Step 3: Remove Dependency (on Apache Modules)

En route to that understanding, I was fairly sure I’d reduced the system Apache to a single responsibility: being a reverse HTTP proxy. To validate that it was no longer serving any other purpose, I turned off most LoadModule directives — even the typical and enabled-by-default ones — leaving only those that prevented Apache from running when I tried turning them off.

Step 4: Substitute Apache with Bozohttpd

I’d been hoping to replace Apache with bozohttpd. Now that I had small, explicit per-site configurations, I could try converting one. The site worked, but the logs were missing lots of basic information. I still think this is where I want to go, but since it’s not a refactoring, I can’t go there yet.

Step 5: Substitute Apache with Lighttpd

I tried converting the same site from Apache to lighttpd, which is a little more featureful than bozohttpd. The site worked, and with mod_extforward enabled, its server logs were indistinguishable from Apache’s. I gzipped the now-retired Apache config to prevent it from being used by mistake while keeping it for reference, updated the site’s crontab entry to start Lighttpd instead of Apache, and rebooted. Bingo!

Step 6: Substitute More Apaches with Lighttpd

I converted a bunch more sites. After doing a few, I figured out how to extract shared configuration. Simpler sites have extremely short config files (just a few lines). More complex sites only define what’s unusual about them.

Step 7: Remove Dependency (on Apache PHP FastCGI)

With a few Apache-powered sites left to convert, I was pretty sure none of them was using PHP. To test this hypothesis, I stopped the php-fpm service. After a week, with nothing broken, I uninstalled it.

With only a few Apache-powered sites remaining, could I return kernel IPC parameters to their default values? Yes, all the Lighttpd and Apache sites ran just fine that way.

Step 8: Get Married

Getting married is the opposite of a refactoring. There’s no internal change, but many callers have new expectations.

Step 9: Substitute Remaining Apaches with Lighttpd

I expected three sites to be relatively tricky to convert:

  1. needed language negotiation to provide English or German content. I didn’t want to futz with it until there was clearly no longer any urgent need for information about the wedding.
  2. needed SSL, which I wasn’t sure whether to proxy at all. Turned out to be easy to proxy because it’s the only HTTPS site I host at present, and it looks like it might continue to not be a big deal if and when I host more.
  3. needed fancy URL rewriting for compatibility with the site’s previous incarnation. I assumed it was going to, anyway. I wound up being able to translate most of its Apache mod_rewrite config to Lighttpd’s expressive conditional redirects, and needed hardly any special-snowflake cleverness.

Once they were converted, there were zero remaining Apache-powered sites.

Step 10: Substitute Apache with Pound

A single Apache instance remained: the system one that was nothing but a reverse proxy to a bunch of Lighttpd instances.

Had I known that’d be its only job, I’d have chosen software designed for the purpose. I knew that now, and chose Pound. On a non-standard port, I figured out how to express a few sites’ worth of reverse proxying in Pound’s configuration language, continued until I’d translated everything in the Apache config, stopped Apache, and started Pound.

Step 11: Remove Dependency (on Apache)

Not a single Apache instance remained. To my knowledge, all sites were operating as normal. After a week, I uninstalled Apache, deleted its corresponding Unix user and group, and gzipped all its config files for reference.


Apache had been serving multiple roles. I brought the number down to zero, then got rid of it. To do that, I…

  • Decoupled Apache (the virtual-host multiplexer) from Apache (the web server)
  • Gave each site its own Apache web server instance
  • Found a suitable replacement web server and converted all instances
  • Found a suitable replacement virtual-host multiplexer and switched to it
  • Turned software off, and left it off for a while, before uninstalling

For human site visitors, all of these steps were genuine refactorings. (Atypical and automated visitors might notice the HTTP header reporting different server software.) For site owners, most of these steps were also genuine refactorings. (In a couple cases, using the shared Lighttpd config required changing the names of log files by a small nonzero amount.)

I replaced one big application with two small ones. Better. Still, could be more better.

Room for improvement

The replacement virtual-host multiplexer (Pound) feels simple, good, and necessary, in the sense that nothing like it is included with the OS. The replacement web server (Lighttpd) feels simpler and better, by far — I understand what it’s doing, my users finally have full administrative control over their own sites, and unlike Apache, this configuration doesn’t require extra system resources — but NetBSD does include a web server, the one I experimented with in Step 4. If bozohttpd did a few more things, then “Replace Lighttpd with Bozohttpd” would be a refactoring, one that could be followed immediately by “Remove Dependency (on Lighttpd)”.

Next steps

I’ve been practicing C. In some kind of cosmic coincidence, next week I’ll be joining a project that’s being developed primarily in C. Hacking on bozohttpd will be good practice. Here’s the incremental sequence of features awaiting my next increment of time and attention, perhaps on tomorrow’s transatlantic flight:

  1. Optionally log to a file (instead of syslog or stderr)
  2. Optionally log more information (say, in Apache’s “combined” format)
  3. Optionally specify a proxy or proxies that can pass an X-Forwarded-For header whose contents we’ll use as the true client source address (for logs, access control decisions, etc.)

Since I believe I’ll be able to stop, I’ll be able to start. It might not be terribly long before I have more progress to share.

Posted Sun Sep 11 18:03:50 2016 Tags: