Welcome back, dear reader, to our ongoing journey through the rich history of DevOps! In the last chapter, we discussed the many movements that led to Agile methodology. Towards the end, we foreshadowed how constant validation would play a role in this chapter.

Now that foreshadowing is coming to light! This time, we’ll be discussing two processes near and dear to our hearts: automated testing and continuous integration.

Automated Testing

We’ve already seen how the Agile movement was inspired by moving away from Heavy Documentation and towards Light Chunks of Work. Everyone was experiencing a collective revelation: risk is unavoidable, so you might as well minimize that risk by cutting it into small chunks — i.e. chew your food before swallowing it, or you’ll choke and die.

As people grew more comfortable with failure, they created places where failure was not only acceptable, it was encouraged. The rise of testing frameworks like JUnit and Cucumber reflected this new attitude toward risk. Tools don’t precede processes — we create them to optimize existing processes. This meant that engineers were spending enough time writing tests to warrant the creation of these frameworks.

Tank.jpg The first automated testing framework was extremely effective but ethically questionable.

Extreme Programming was the primary influence behind this Zest for Tests. Kent Beck and Ron Jeffries had championed test-driven development with a convincing argument: “If bugs are going to happen (and they will), engineers should fix at least some of them while they still remember the code.” Because code evaporates from an engineer’s mind faster than a drop of baijiu.

Writing tests before the code forced engineers to think about how they wanted software to work. In many ways, this practice served the same purpose as the exhaustive documentation of the Waterfall years: tests specified how things should be. But automated tests went one step further by enforcing these requirements on each code change, loudly complaining when Something Was Not Right.

Engineers discovered that, by running tests more frequently, they increased the odds of catching bugs and preventing catastrophes. That meant checking as you go instead of at the end. This might seem obvious today but, at the time, it was a fundamentally different way of developing software.

And it’s from this bubbling, primordial soup of testing that continuous integration was born.

On the Origin of Continuous Integration

The DevOps Timeline isn’t as clean as we’d like it to be.

Rather than a discrete progression from the Agile Era to the Age of Continuous Integration, the two concepts are actually contemporaries. The phrase continuous integration (CI for short) is actually older than many of Agile’s ancestors. It was coined in 1994 by Grady Booch, a man of considerable academic and mutton chops.

Booch is most (in)famous for developing the Unified Modeling Language (UML) with two colleagues: Ivar Jacobson and James Rumbaugh. UML is a great example of the overzealous design phase from which the Agile Alliance tried to distance itself, but otherwise has no place in this story. Begone, rabbit hole, though your depths are tempting!

Booch covering his mutton chops in surprise.

The phrase itself appears in Booch’s book, Object-Oriented Analysis and Design with Applications:

“The needs of the micro process dictate that many more internal releases to the development team will be accomplished, with only a few executable releases turned over to external parties. These internal releases represent a sort of CONTINUOUS INTEGRATION of the system, and exist to force closure of the micro process.”

While the bolding and capitalization are my doing, the rest is directly from Booch’s head. He also writes that testing should “be a continuous activity during the development process”, so from early on, testing and continuous integration were closely intertwined.

Pay close attention to Booch’s choice of words: he describes the “needs of the micro process”, and this is a recognition of the pace of software development. He’s not discussing normal-sized or macro processes — no, these are micro processes because the slices of delivery were being cut ever more thinly.

Spot the Difference

If you’ve read our discussion of feedback loops (you better have!), these words and thoughts should be ringing all sorts of bells. The Agile movement itself could be described as a grand reduction of feedback loops in software. Continuous integration — while more process-oriented than Agile’s various tactics — is a natural extension of this reduction.

When done well, continuous integration acts as a kind of unending course correction — a formalized application of the belief that uncertainty is the one certainty. While it has much in common with its Agile cousins, continuous integration differs in scale. The Agile movement provides a bundle of tactics that makes an organization more responsive; continuous integration is a strategic shift that fundamentally changes an organization’s culture.

But these distinctions quickly become blurry. When Kent Beck and Ron Jeffries created Extreme Programming in 1996, they officially adopted continuous integration as one of its twelve core practices. They even took it further, advocating multiple integrations per day.

So: Agile and continuous integration were contemporaries, one didn’t necessarily lead to the other, and continuous integration is not just another Agile tactic — even though it is an application of Agile principles. All of this ambiguity is an unfortunate side effect of discussing recent events: we haven’t really had time to agree on a single narrative yet!

But there was one thing everyone could agree on: CI was a Good Thing — good enough to build tooling around the Thing to make it repeatable and scalable.

ThoughtWorks and CruiseControl

In 2000, one of the founding members of the Agile Alliance became an internal advocate for continuous integration at ThoughtWorks. His name is Martin Fowler, and he’s rather famous. Fowler points out that continuous integration isn’t conceptually tied to specific software; developers can practice CI without a tool dedicated to it. But — as with any home-rolled solution — personalization comes at the cost of time, money, and the ability to build a network of users.

So practicing CI can be made easier with a server built specifically for daily builds and continuous testing. And in 2001, another ThoughtWorker named Matthew Foemmel built this and dubbed it CruiseControl. The tool is open-source and still around, though there are fewer maintainers these days.

CruiseControl was important. It helped bring a best practice into reality, and its public nature encouraged adoption across the industry. It didn’t have a ton of responsibility: watch for changes in an application’s codebase and, if necessary, create a new build and run it through a suite of tests to verify that everything actually still works.

Rickle.png What happens when you don’t continuously integrate.

Hudson and Jenkins

While CruiseControl was a relatively simple tool, it represented a commitment to both software and chronological integrity. Soon, other contenders were sprouting up around the concept like daffodils. Hudson, another CI tool written in Java, was released in 2005 but didn’t really start to overtake CruiseControl until 2008.

Then! As all good stories are wont to do, there was a Dramatic Dispute. Hudson’s author, Kohsuke Kawaguchi, had been working at Sun Microsystems, which Oracle purchased in 2010. Like CruiseControl, Hudson was free and open-source, so when Oracle announced its plans to trademark and commercialize it, the Hudson community fled the scene in 2011 under the new name of Jenkins which continues strong to this day.

(Best) Practice Makes Perfect

There are many places to read about the fallout of that split, so we won’t cover them here. Instead, let’s take a step back and discuss how** these various tools (and disagreements) represent the **industry’s rapid acceptance of a best practice.

In just a decade, developers formulated, proposed, implemented, and commercialized a suggested process. The fact that Oracle was willing to put up a fight indicates that the idea — and its implementation — have value. And that value has percolated across the software world until it’s become so well-ingrained that the question has shifted from “Are you continuously integrating?” to “When and how often are you continuously integrating?”

For a while, Facebook’s motto was “Move Fast and Break Things”. As the software industry has matured, engineers have realized that you can’t actually move that quickly if everything is breaking. That’s why Facebook changed its motto to “Move Fast With Stable Infra”.

Continuous integration is the first step on that path. Catching problems before they get out of hand demonstrates a shift in the treatment of risk. It’s the logical next step on the agile escalator, and the reason it’s an escalator in this metaphor is because the journey never ends.

In theory, at least. In practice, this series is going to end in the next part, when we discuss continuous delivery and deployment!