Killing Your Build Servers For Fun And Profit, But Mostly Profit

snowflake.png

There’s an infamous quote that circulates among writers: “Kill your darlings.” How violent and succinct! And, as it turns out, quite applicable to software delivery.

The quote has been attributed to many eminent writers, from Faulkner to Wilde, but its earliest uttering was from Sir Arthur Quiller-Couch’s 1916 Cambridge lecture series. In it, good ol’ SAQ-C said:

“Whenever you feel an impulse to perpetrate a piece of exceptionally fine writing, obey it — whole-heartedly — and delete it before sending your manuscript to press. Murder your darlings.

It is hard to say goodbye to good writing, but it is this ruthless pruning of the “outstanding” that results in a piece worth reading. And there’s an obvious parallel in the world of software development: rip out code that — while showing great evidence of your cleverness — is Baroque in both style and function.

Kill your darlings.


Let’s say you’ve poured hours into setting up a build server: it’s a beautiful thing, brimming with custom configs and soaked in scripts. Everything works flawlessly. You are a DevOps demigod and wield your power with staggering precision. When tests devour system resources, you spin up additional boxes with panache, reveling in a provisioning script you wittily named whoo.sh.

This feeling is short-lived. Requests from developers start trickling in, benign asks like, “Could you install v4.2.0 of the devise gem for me? I need it.” or “This test ran on my machine, but broke in the build; can you give me SSH access? I need it.” You are patient, granting wishes like a gracious genie.

Soon, it’s Time to Scale, and that trickle of demands becomes a deluge. Innocent whoo.sh trembles at the prospect of hourly maintenance. You realize that your build servers have “minor differences”: a few hundred CPU cycles here, 16GB of RAM there. You ask whoo.sh to mediate, reconciling and special casing — but like all minor differences, these fester and eventually explode at the family dinners of deployment.

You find you’ve become a gatekeeper for needy developers, passing out credentials like Halloween candy when they inevitably need to debug their builds. You are a dentist, removing the plaque of corrupted installations from the molars of your server fleet. Developers develop a dependency on you managing dependencies.

This is not the life you thought you would be living. Your bespoke solution has brought your efficiency down and your temper up. You retreat into your (bash) shell, pull your hoodie over your head, and fall into a fitful sleep, where the frightful forms of site outages pursue you across a nightmarish landscape.

“But hark!” says Sir Arthur Quiller-Couch from the grave, “awake and remember my teachings!”

You wake up suddenly, your mission clear: you must kill your darling build servers.

Feverishly, you scan the web for an answer. A solution gleams from the guts of Google like a Vegas billboard. You click on it.

“LXC containers for each build?” you ask incredulously. “It’s so simple.” A single tear forms, rolls down your cheek, and splashes onto the keyboard.

Down you dive, deeper and deeper into an ocean of documentation. Of course, you’ve heard of containers before — everyone has. “Containerize your authentication, containerize your geolocation, containerize your kids…”

But this makes more sense. Your hands shake as you realize every build is destroyed after use, like a CONFIDENTIAL document. Cruft doesn’t even have a chance to accumulate: it just gets blown away every time a developer kicks off a build. It’s like… continuous disintegration.

And developers won’t be able to play ping-pong while they wait for you to provision a new box; you’ll just add more containers and deny them their fun.

The sugar-free icing on the cake: each developer can SSH into an isolated environment to debug their builds without bothering you. Less developer babysitting means more time for important tasks, like doing actual babysitting for your actual children.

A Slack notification jumps up and down excitedly at the bottom of your screen. You click and a message from a developer unfurls:

unspecified.png

… you reply, pulling Slack’s Do Not Disturb mode over your head like Batman’s cowl.

You count your various build servers — ten, twenty, thirty — rub your hands together with glee, and prepare to exorcise your dæmons.

You hook up your company’s main repo, an ambitious app called “ZombiePosts” that resurrects controversial topics from ‘90s forums. Just linking the repo triggers a build on the dashboard and goosebumps on your skin.

Watching with wild, bloodshot eyes, you see each phase of the process unfold in mint-fresh green. It is a beautiful sight, yes, but could be faster.

“Ahead, warp factor eight,” you bark. The systems respond, scattering tests across newly-added containers like feed to so many chickens. Parallel chickens.

Then you wait.

You half-expect the build to emit a domestic “DING” when it’s done. But instead you see a perky web notification: “Yay, your tests passed!”

You don’t normally trust such forceful optimism, but when you glance back to your build, it shines brilliantly, an Emerald City of passing tests. If you weren’t so dehydrated from the heat of the battle, you would shed more tears now, but your eyes are as dry as your code.

But you have to express your emotions somehow. So, in the overused #general channel, you type “@everyone: the nightmare is over.”

You hesitate; there is no coming back from an “@everyone”.

A ghostly sigh interrupts your thoughts. “My friend, you killed your darlings,” whispers the shadowy form of Sir Arthur Quiller-Couch.

“I know, Arthur,” you croak. “I know.”

Then you press Enter.


This is a work of satire. But CircleCI does support parallel testing, clean containers for every build, and extensive documentation. You can learn more and sign up for your own CI epiphany here.