Three tools, three temperaments, one war

Previous issue: a physicist told machines to keep their own promises, and then watched a more usable dialect take the room. This week, that one quiet conviction becomes a loud, crowded, occasionally bitter market.

In 2008, a man named Luke Kanies spent something like ninety thousand miles in the air, trying to convince the world it had been configuring its computers wrong.

Picture the life of it. The middle seat, the rental car, the stale ballroom where forty system administrators have wandered in partly for the free coffee and partly because the talk is between them and lunch. Kanies, a working sysadmin himself and no natural showman, would stand at the front and explain an idea that sounded, depending on the listener's mood, either obvious or faintly heretical: stop logging into your servers and changing them by hand. Write down what you want each server to be, in a file, and let a program make it so.

This was not an easy sell. The people in those rooms were good at their jobs, and being good meant being fast at the keyboard, trusted to fix the broken thing at two in the morning. Kanies was asking the most skilled manual workers in the building to describe their craft so precisely that a machine could perform it. Which is a strange thing to ask anyone, and a particularly strange thing to ask of people whose value was bound up in being the one who knew.

He flew the miles anyway. Within a few years, the quiet conviction of one traveling sysadmin had become a war.

Here is the sneaky part. If everyone agreed on the big idea, that you should describe your infrastructure as code rather than build it by hand, then what was left to fight about?

Everything that matters in practice, it turns out. Agreeing that infrastructure should be code is like agreeing that buildings should have foundations. It tells you almost nothing about how to actually pour one. The combatants had all accepted the declarative grammar from Issue #2. What they disagreed about, ferociously, was the language you should write the description in, and underneath the language, a deeper and more human question:

How much power should the tool give you, and how much should it take away to keep you safe?

You can feel the full weight of that only at a specific hour. It is three in the morning. Something is broken. You are not at your best, because nobody is at three in the morning, and you are reaching for a tool to fix a server thousands of people depend on. Do you want a tool that speaks a small, strict, deliberately limited language, one that won't let you express anything too clever and therefore won't let you do anything too catastrophic? Or do you want the full power of a real programming language, every loop and conditional, so you can express exactly the surgical thing you mean, at the price of also being able to express exactly the wrong thing with total fluency?

There is no correct answer. That is the point.

Three people, three temperaments, built three tools that each answered the question differently. Each tool is, in a real sense, a personality rendered in software.

Kanies wanted rigor. Puppet (2005) gave you a custom language built for exactly one purpose: describing the desired state of a machine. Strict, opinionated, but legible, approachable enough that a working administrator could read a Puppet file and understand, more or less at a glance, what machine it described. Rigor with a human face.

Adam Jacob wanted expressiveness. Chef (released 2009) wrote its configurations, called "recipes" and bundled into "cookbooks," in Ruby itself, a full general-purpose language. Jacob's wager was that operators were, or could become, real programmers, and that they'd rather have genuine power than be fenced into a limited dialect. If you wanted a loop, you had loops. Chef trusted you. It handed you the sharp tools and assumed you were an adult.

Michael DeHaan wanted simplicity. Ansible (2012) was built around asking for as little as possible. Where Puppet and Chef installed an "agent" on every machine, Ansible installed nothing. It reached out over SSH, the same connection every administrator already used, and pushed your configuration. If you could SSH to a box, you could manage it today, with no preparation. And the configs were written in YAML, a plain, almost punctuation-free format readable by humans who were not programmers. You wrote what looked like a structured to-do list in near-English, and Ansible carried it out.

Rigor, expressiveness, simplicity. The war was not good versus evil. It was three flavors of good, and the operators of the world had to choose.

Under all three sat one shared idea, the quiet engine of the entire category. It has an intimidating name and a beautifully simple meaning: idempotency.

Picture a light switch that does not toggle but only ever turns the light on. Press it once, the light is on. Press it again, still on. Press it forty more times in a panic, it remains, serenely, on. You cannot make things worse by repeating yourself. That is idempotency, and it is the single most important property a configuration tool can have.

Contrast it with its absence. Imagine a thousand toggle switches across a thousand servers, and you're not sure which are currently on. Send "toggle" to all of them and you've guaranteed you now have no idea what state your fleet is in. That is the imperative nightmare: the world of the snowflake and the drift.

When you write "this package should be installed," you are pressing the on-only switch. Run it against a machine that already has the package, and the tool does nothing. Run it against one that's missing it, and the tool installs it. Run it a hundred more times, and after the first the answer is always the same calm nothing. Run it twice, get the same box. Run it a thousand times, get the same box. The boredom was the breakthrough. In a field where excitement usually meant an outage, a tool that made change boring was a tool that could let you sleep.

For a while the war looked like it might be won on power. Puppet and Chef were the serious tools, the ones large organizations adopted, the ones with conferences and certifications and résumé lines. They were, for several years, the establishment.

And then Ansible, the latecomer, the deliberately simple one, spread like water finding cracks.

It spread because of exactly the thing its rivals had bet against. The agent you didn't have to install. The YAML a newcomer could read on their first day. The SSH connection that worked on machines you hadn't prepared. Each lowered the barrier by another inch, and inches, multiplied across a whole industry of busy people, become a flood. An engineer could discover Ansible in the morning and have it managing real servers by the afternoon. A great many people will choose the thing that asks for almost nothing, especially when they are tired, especially when they just need the server to work before lunch.

But here is where the easy story lies to you. The temptation is to say simplicity won, that the war ended with the agentless YAML tool standing over the bodies of its rivals. False, and false in an instructive way. Puppet did not die. Chef did not die. The war did not end with a surrender. It diffused, the way real arguments do, into an uneasy coexistence in which different teams quietly used different tools for different reasons and mostly stopped shouting.

The most trustworthy critics here are the protagonists themselves. Kanies said the quiet part out loud about his own creation: some users, he admitted, "are frustrated by how simple it is and wish they had a full Turing-complete language like Ruby." Read that carefully. It is a confession from the architect of rigor that rigor has a cost. The guardrails that keep you safe at three in the morning are the same guardrails that, at three in the afternoon, stand between you and the perfectly reasonable clever thing you're trying to express.

And Ansible's accessibility curdles, under pressure, into the most famous critique of the era. Readable YAML works beautifully right up until your configuration needs to do something that isn't simple. The moment you need a loop, a condition, a computed value, you discover YAML was never a programming language. It was a data format wearing a programming language's clothes. The industry coined a weary phrase for the result: "YAML as a programming language," said in roughly the tone you'd use to describe a house someone tried to build out of envelopes.

Notice the shape this makes, because the shape is the lesson. We do not have a winner. We have a pendulum. The field swung from custom-DSL rigor, toward full-power expressiveness, and then hard toward YAML approachability, and the same complaint that pushed people away from the strict language ("too limited, give me a real language") is the mirror image of the one that would later push them away from YAML ("this isn't a real language, stop pretending it is"). Every gain in accessibility is a loss in power. Every gain in power is a loss in safety. A tool can sit anywhere on that arc, but it cannot sit everywhere. The strength and the weakness are not two facts about a tool. They are one fact, seen from two distances.

Something larger was happening alongside the tools, and it's why this chapter is titled the way it is.

For decades a wall had divided the industry: the people who wrote software, the developers, and the people who ran it, operations. Two tribes, two cultures, two sets of grievances. Developers threw code over the wall; operations kept it running and blamed the developers when it broke. "It works on my machine" was, in part, a border dispute between these tribes.

What configuration-as-code did, almost as a side effect, was give both tribes the same medium. Once a server's configuration lived in a text file, a file you could store, review, and run, operations work started to look like development work. It went in the same kind of repository. Got reviewed the same way. The operations people were becoming programmers, and the developers increasingly had to understand the systems their code ran on. The wall came down because the two kinds of work were converging into one kind of work expressed in one kind of artifact.

That shift got a name: DevOps, crystallized at an event called "devopsdays" in Ghent in 2009, and rehearsed a year earlier in a now-legendary talk titled "10 Deploys a Day." It was not a product launch. It was a name for a merger the tools had already begun.

And that is the quiet thesis-level point. Codifying the work changed the people. The wall didn't fall because anyone passed a rule. It fell because the work itself changed medium, became text, became code, and a shared medium forced a shared culture. Turn a kind of work into code, and you don't only automate the work. You reshape the humans who do it, their roles, their tribes, their sense of who they are.

Code, it turns out, configures more than servers.

Chapter 3 of a twenty-chapter argument. The pattern of this chapter is the one to hold onto, because it recurs at every scale the book climbs to: whenever some category of work becomes codeable, you don't get one clean tool and universal agreement. You get a war, the good kind, a contest between the people who want maximum power and the people who want maximum accessibility. The config management wars were just the first time it played out in the open, with conferences and acquisitions and partisan T-shirts. The same pendulum will swing when we try to write documents as code, teams as code, whole organizations as code.

→ Read more and follow the series at workascode.ai

Next issue: even as the partisans argued about YAML versus Ruby, the ground beneath them was being pulled away. A company in Seattle had begun renting out servers that didn't physically belong to you, that you summoned and dismissed with a command. When the machine itself becomes a sentence you speak, the question stops being "how do I configure this server" and becomes the far stranger "how do I describe an entire data center that does not yet exist?" Issue #4: the cloud ignites.

There was never a winner of the config-management wars, only a moving truce between the engineer who wants the machine to do exactly what she says and the one who just wants it to work before lunch.

See you in Issue #4, Eleodor Sotropa

You're receiving this because you follow the future of programmable work. Subscribe · Forward to someone who has strong opinions about YAML.