At the 2009 O’Reilly Velocity conference, John Allspaw and Paul Hammond, who at the time were engineering managers at Flickr, gave a talk about the best practices that allowed Flickr’s DevOps organization to deploy 10+ times a day. That now-famous presentation sparked a lot of discussion about the optimal number of builds to deploy daily – and how to increase deployment frequency.
But, as we explain in our earlier post about workflow lead times, quantity and speed don’t always equate to quality. Deploying code is essential, and high-performance DevOps teams do, in fact, deploy code much more frequently than their low-performing peers. However, there’s no magic number of deployments that guarantee that you’re on a high-performance team; the number of deployments depends on the software you’re developing.
Learn how to find and monitor engineering metrics for team success in this post.
As we share in our recent report, The Data-Driven Case for CI: What 30 Million Workflows Reveal About DevOps in Practice, deployment frequency, along with other key metrics like lead time and change fail percentage, can be improved with continuous integration and continuous delivery (CI/CD). Using run data from 30 million workflows through CircleCI, we find that the numbers support key DevOps metrics presented in sources like the 2019 State of DevOPs report. In this post, the second in a four-part series, we’ll take a deep dive into deployment frequency.
The connection between deployment frequency and the ability to deploy at will
Deployment frequency indicates how many discrete units of work are moving through the development pipeline. For our report, we measured how often a team kicked off a workflow. Automation, which you get when you adopt CI/CD, allows even complex codebases to be deployed automatically. This is essential to improving deployment frequency. In an automated software pipeline, things like hotfixes are available to your users almost immediately, and go through the same rigorous testing as any other update.
What’s important about deployment frequency is that it serves as an indicator of a team’s ability to deploy at will. If your team has confidence in its CI pipeline, and can automate deployments, they don’t have to deal with manually validating code, a big stumbling block for more frequent deploys. Using automation, testing can be performed on every commit, and failed build notifications allow for immediate visibility into failures. And with error information delivered to you quickly, your DevOps team can push out changes faster.
To understand how observed development behavior compares with industry standards, we looked at CircleCI data from over 30 million workflows observed between June 1 and August 30, 2019. The workflows represent:
- 1.6 million jobs run per day
- More than 40,000 orgs
- Over 150,000 projects
Here’s what we found:
- 50% of DevOps organizations start six workflows per day across all projects
- At a per-project level, 50% of projects have three workflows started per day
- In the 95th percentile, these numbers rise to 74 workflows per project, and 250 workflows across all projects
Given the high profile of Allspaw and Hammond’s 10-year-old presentation about deployment frequency, we would have expected to see many of today’s teams deploying with very high frequency. Instead, we did not see many teams deploying tens of times per day or more. While this behavior does exist, it is the exception, not the rule.
What’s the takeaway from our data?
Even though we don’t see many orgs hitting the supposed magic number of 10+ deployments per day, we can observe the impact of CI/CD on deployment frequency. According to the Accelerate State of DevOps 2019 report:
- Elite teams deploy several times a day
- High-performing teams deploy anywhere between once a day and once a week
- Low-performing teams deploy between once a month and once every six months
These findings align well with our run data showing that the average CircleCI user runs six workflows a day. Since a DevOps team using CircleCI can deploy whenever needed, the number of deployments becomes a reflection of an organization’s choices, not necessarily a reflection of high performance. The frequency of testing and integrating code, and the speed of a team’s recovery when they encounter errors, are better indicators of high-performing DevOps teams than deployment frequency.
Why CI helps you consistently build the best software
As our data shows:
- Teams using CircleCI are incredibly fast: 80% of all workflows finish in less than 10 minutes
- Teams using CircleCI stay in the flow and keep work moving: 50% of all recovery happens in under an hour
- 25% of failures on CircleCI recover in 10 minutes
- 50% of failures on CircleCI recover in one try
While we’re still examining the question of whether CircleCI enables teams to become high-performing – or whether high-performing teams have simply made a habit of choosing CircleCI – we can say with confidence that teams committing to the practice of continuous integration can produce the best software consistently and at high velocity.
We can also say with confidence that we know implementing CI isn’t easy – but simply starting the process will yield benefits, even if you’re not deploying a thousand times a day or investing millions in infrastructure.
In the next two blog posts on our workflow data, we’ll share insights on more key DevOps metrics: mean time to recovery and change fail percentage. In case you missed it, here is our first post on lead times.