(This is part two of a four-part series. Read parts one, three, and four here.)
Welcome to another chapter in the insight-rich story of DevOps!
Last time, we discussed why the history of software development is important and how waterfall development fit into that narrative.
In this chapter, we’re going to explore how (and to what extent) engineers iterated on the waterfall model by inventing Agile software development. Instead of trying to control change by locking down phases of development (like the waterfall method), Agile methodology is about embracing change. Risk is mitigated not by designing the perfect plan, but by cutting projects into small chunks and adapting on the fly.
Agile predecessors
While it would be really convenient if waterfall development led directly to Agile development, the actual timeline isn’t so straightforward. Even in Winston Royce’s 1970 paper, there was already a general awareness of the need for iteration and lightweight processes. This awareness only grew as developers discovered weaknesses in existing workflows.
These workflows were heavy, inflexible things. Software was still treated as something to be manufactured, like hardware. Behemoths like Microsoft focused on planning and eliminating surprises during development. Perfectionism was encouraged, resulting in astronaut architecture and Big design up front.
Generally, Agile methodology was a reaction to overzealous documentation and low thresholds for change. For Agile advocates, the software industry was too focused on process and planning, to the detriment of both products and customers. While features languished in bureaucratic, corporate waiting rooms, everyone suffered: developers couldn’t ship their creations, products fell behind the competition, and users either had to use outdated technology or flee to a competitor.
All this pressure to improve led to a bevy of workflow experiments in the 1990s. The results of these experiments were the precursors to Agile methodology and shared common beliefs around speed, risk, and chaos.
Rapid application development
At IBM, James Martin created Rapid Application Development, which emphasized process and prototyping over the excessive planning of waterfall development. He consolidated his ideas into a 1991 book, Rapid Application Development (RAD).
RAD gained fans quickly, and prototyping became increasingly viable across the industry. But its fans disagreed on the “correct” way to RAD, in part because of the intentionally vague instructions left by Dr. Martin. Businesses were cautious about embracing a fad that might sacrifice quality for speed.
Dynamic systems development method
In 1994, the Dynamic Systems Development Method (DSDM) emerged to unify the many versions of RAD. Unlike its predecessor, DSDM was spearheaded not by an individual, but by a consortium of motivated vendors and experts. And instead of just outlining a process, DSDM got its hands dirty by defining explicit techniques and project roles.
Two of those noteworthy techniques are timeboxing and MoSCoW prioritization. Timeboxing cuts a project into chunks that have resource caps — on both time and money. If either resource begins to run out, efforts are focused on the highest priorities, defined by MoSCoW. While the actual acronym is questionable, MoSCoW’s intents are sound. Look at what those capital letters stand for:
-
Must have
-
Should have
-
Could have
-
Won’t have
By grouping requirements into these four levels, engineers always know exactly which ones to drop when pressed for time or money.
Both timeboxing and MoSCoW are concerned with time, safeguarding against the software industry’s tradition of perfectionism. By implementing constraints on how much work can be done and establishing frameworks for the value of that work, DSDM kept people moving.
Object-Oriented Programming and more
But folks weren’t done iterating! In 1995, Kevin Schwaber and Jeff Sutherland, went to a research conference on Object-Oriented Programming, Systems, Languages & Applications. The acronym for this is OOPSLA, which sounds accidental but is actually very intentional.
There, Schwaber and Sutherland presented a jointly-authored paper on a new process called Scrum. Scrum was yet another framework for developing software and focused on collaboration and flexibility. The term comes from rugby, where both teams restart play by putting their heads down and focusing on getting the ball.
Scrum is one of the first philosophies that explicitly emphasized agility and empirical process control. All of Scrum’s values revolve around a single principle: uncertainty. Are you certain customers will like this? Are you certain you understand everything about this project?
True certainty, argue Scrum adherents, is impossible to achieve — an asymptote of productivity. Where waterfall attempted to define everything ahead of time, Scrum abandoned that fantasy in favor of smaller batch sizes. The goal is to hone the delivery process instead, focusing on adaptation and emerging requirements. This is the essence of Scrum and drives its tactics.
Those tactics include familiar routines like sprints, daily standups, and retrospectives. All of these are meant to channel work into focused bursts of productivity. The regular check-ins provide opportunities to course correct and aim energy at emerging priorities.
One important tool of Scrum is the backlog. There are usually multiple backlogs at any given point: one for the product generally, and then a backlog per sprint. These backlogs give the team an efficient method for prioritizing work. These artifacts are so important to proper Scrum execution that whole industries have grown up around them. Pivotal Labs and Atlassian’s JIRA are just two of the companies creating issue trackers.
Extreme programming
Extreme Programming (XP), coined by Kent Beck in March 1996 advises engineers to gather best practices and ratchet them up to an extreme level. Beck was working on a payroll project with Chrysler when he devised the basic tenets of XP. While the project was canceled, the methodology survived.
For example, if some testing is good, then more testing must be even better. This translates to test-driven development, where tests describing behavior are written before the code is developed. This practice not only increases the total number of tests in an application, but also changes how the code itself is written.
Another example involves optimization of feedback loops. Advocates of XP argue that a ruthless focus on short feedback loops increases the odds that you’re developing the right product. Extreme programmers should write code for the present and receive feedback on that code as quickly as possible.
This commitment to feedback manifests in practices like pair programming, where developers write code in two-person units. By having to articulate what they’re writing, engineers write better code and find bugs before they happen. Teams also commit to frequent releases, often manifesting in processes like continuous integration.
Exploring agile themes
If you’ve noticed a theme here, you’re not alone. While these philosophies emerged independently, there was a general awareness of the need for speed. But along with that collective recognition, there was also a genuine lack of common language: words like evolutionary and adaptive were being tossed around, but these had different nuances depending on which methodology you were using.
There were simply too many cooks in the kitchen, and some of these cooks hadn’t even met each other! Some of these thought leaders had crossed paths at other conferences, but in September of 2000, Bob Martin sent out an email to gauge interest, and titled it The Agile Manifesto.
The manifesto
On February 11, 2001seventeen developers — many of them representatives of the aforementioned methodologies — met to organize their thoughts on what it meant to be “Agile”. The Agile Manifesto isn’t very long, consisting of twelve principles total. These principles are based on four values that summarize how Agile developers should make decisions:
- Individuals and interactions > processes and tools
- Working software > comprehensive documentation
- Customer collaboration > contract negotiation
- Responding to change > following a plan
The Agile Manifesto consolidated several existing methodologies: from Extreme Programming, they lifted customer satisfaction and collaboration; from Scrum, they incorporated the introspection of standups and retrospectives; and added the … pragmatism from Pragmatic Programming.
The Agile Manifesto was not a new methodology, but instead a collection of existing methodologies. The “Agile Alliance” brought all the best parts together, hoping that the whole would be better than its various components.
The Agile Manifesto was not a new methodology, but instead a collection of existing methodologies. The “Agile Alliance” brought all the best parts together, hoping that the whole would be better than its various components. The idea of Agile can sometimes distract from the Agile methodology itself. Ultimately, Agile is nothing more than a collection of engineering principles, created to help software teams make practical decisions more quickly. It is not an exhaustively-defined process or magical recipe for success.
Conclusion
Even if Agile is just a collection of engineering principles, does that make it unhelpful or impractical? Not necessarily. Agile is a movement. And like all movements, it’s easy to become emotionally invested without understanding everything about it. It doesn’t help that “agile” is also just a normal word that means something simple: nimble, flexible, adaptable.
The larger theme here is that the mission of this gathering was to help people change the way they thought about developing software. Instead of a product to be manufactured, code should be an art to be crafted. That subtle difference implies a host of alternate strategies, such as timeboxing, pair programming, and TDD.
Waterfall was slow and rigid. It couldn’t adapt to change, so engineers tried to control it. But that wasn’t responsive enough to deliver quality software customers actually wanted, when they needed it. Agile methodology was a response to this weakness, and its advocates realized that they could never completely eliminate risk or uncertainty from a project. Instead, they focused on containing risk through rapid development and constant validation that their work was heading the right way.
This idea of constant validation will be expanded upon in the next chapter in the history of DevOps: continuous integration and delivery!
Sources
