The night one tired engineer deleted the internet

It was just before eleven at night when the engineer typed the command.

GitLab, January 2017. A long day, a flaky database, eleven terminal windows open at once, each a small green portal into a different machine. He needed to wipe a replica's data directory so it could re-sync from the primary. Routine. Slightly tedious. He typed rm -rf, the three most quietly violent characters in the Unix language, the ones that mean remove, recursively, by force, and do not ask me whether I'm sure.

He ran it.

Then he noticed the command was taking too long. The replica should have been nearly empty. This directory had three hundred gigabytes in it, and the gigabytes were disappearing, and somewhere between one heartbeat and the next he understood which of his eleven identical windows he had actually been typing into.

It was the primary. The live, beating heart of GitLab.com. The place hundreds of thousands of developers kept their code and their conversations. Gone, by the hand of one of the people whose job was to protect it.

Then came the part nobody tells you about. GitLab had five separate backup mechanisms. Defense in depth. The responsible architecture you'd feel good about on a whiteboard. They reached for the first. Broken. The second. Broken. One by one, all five turned out to be present, in place, and completely useless. The nets were all hanging there, and every one had a hole nobody had checked for.

What saved them was luck: a six-hour-old snapshot someone happened to take for an unrelated reason. Six hours of real customer data was gone for good.

Here is the strange thing. A computer is a machine built to do the same thing twice. It is deterministic almost to a fault. And yet for most of computing history, the humans operating these relentlessly repeatable machines could not reliably repeat themselves. They couldn't guarantee the server built on Tuesday matched the one built on Thursday. They couldn't answer the simplest, most terrifying question in operations:

If this machine died right now, could we make another one exactly like it?

For decades, the honest answer was no. Not quickly. Not confidently. Not without one particular person who happened to remember.

This is the manual inferno. And the comfortable version of the story, the one where this all happened in some primitive past, is a lie. GitLab was 2017. Knight Capital, the Wall Street firm that lit $440 million on fire in 45 minutes because new code reached seven servers and not the eighth, was 2012. Amazon taking down a chunk of the internet with a single mistyped command was 2017. These were sophisticated, automated, deeply competent organizations. The inferno wasn't something they failed to escape. It lived inside their automation, waiting.

That's the secret the easy story hides. Automation didn't prevent these disasters. Automation is what made them enormous. Each began with a normal person doing a normal thing slightly wrong, and powerful systems instantly amplified the mistake to catastrophic scale.

This is why GitLab is the hero of the chapter and not just its cautionary tale. Faced with a self-inflicted wound, every safety net exposed as theater, hours of data permanently lost, they did something almost nobody does.

They did it in public. They opened a live document and narrated the recovery as it happened, edits visible to the world. They published a blameless postmortem documenting every broken assumption. They reportedly even live-streamed the climb out of the worst day of their operational life.

The lesson isn't "manual work is chaos and code is order." That's too clean. The real lesson is harder and more useful: codified systems fail too, just in larger and more surprising ways, and the thing that turns failure into progress is not better tooling alone. It is transparency. Making every change to a system legible: written down, reviewable, reproducible, and owned out loud.

That insight is the seed of Infrastructure as Code. And it is the seed of something much larger.

The disasters above share a single root. In every case, the true state of a system lived somewhere it could not be read, reviewed, or reproduced. It lived in the accumulated edits of a hand-tended server. In the difference between seven machines and an eighth. In the false belief that five backups existed when zero did.

The radical idea that organizes everything: what if the work itself were written down, not as documentation describing the work, but as the literal, executable definition of it? The file becomes the source of truth. It can be reviewed before it runs. Versioned. Run a thousand times with the same result. No snowflakes. No tribal knowledge, because the knowledge is the file, and the file does not quit, retire, or forget.

That's Infrastructure as Code in one breath. But the question it raises refuses to stay in the server room. Deployments are work. So are documents, onboarding, approvals, the design of teams, the running of whole organizations. All of it, today, is mostly snowflake work: done by hand, held in heads, drifting quietly, impossible to repeat.

What happens when we learn to treat that like code too?

Whether that's liberation or hubris is the argument of my new book.

A twenty-chapter journey from a server-room fire to the edge of agentic organizations. Not a confident prophecy. The other kind of book: one that hands you the instruments to tell a true number from a verified one, a demonstration from a forecast, a tool from a theorem.

The whole argument, compressed to five words: Described. Versioned. Tested. Owned. Improved.

→ Read more and follow the series at workascode.ai

Over the next four issues, I'll condense the chapters that built the discipline: a physicist who told machines to keep their own promises, the config-management wars, the cloud turning servers into sentences, and Terraform's attempt to describe the entire world in a file.

The machines were never the inferno. We were. Or more precisely, our silence was.

See you in next Issue.

Eleodor Sotropa

You're receiving this because you follow the future of programmable work. Subscribe · Forward to someone who's ever typed a command into the wrong window.