Think back to your last one-on-one with your manager.
Did you go for a stroll about town? Did you get coffee? Do you get coffee every single time you have a one-on-one? I hope not, that’s a lot of coffee. It probably made you jittery, didn’t it? Made it hard for you to form words that make sense.
You had a perfect argument for a new tool or process, but it died young on your tongue because of the toxic levels of caffeine coursing through your veins. Yes, sometimes words can be hard. And soft, too. Actually, words come in a variety of textures, and that is the beauty of language.
But I’m not here to talk at you about how attractive language is. I’m here to show you how you can use language to convince your boss that continuous integration and delivery are actually concepts worth adopting within your organization.
Here is a Strongly Worded Email you can send to your boss with (minor) edits to convice them that CI/CD is the future.
Subject: A Case for CI/CD
So, I know I’m supposed to be writing up a post-mortem about last night’s outage. And I have some of that draft done; marketing is getting really excited because there’s nothing as magnetic for views as a long, detailed explanation of how we fucked something up. This is because engineers love rubbernecking and schadenfreude more generally.
But let me set the scene for you:
We used to push code all the time. We were Silicon Valley’s darling, announcing features, integrations, and plugins during our weekly town halls. We had our customers accustomed to daily dopamine hits with the latest and greatest, and they couldn’t get enough because there was always more.
Those days are behind us.
We haven’t pushed a new feature in four months. There are horrific areas of the codebase that no one understands because Leroy never explained them before renouncing all ties with technology and running naked into the wilderness. Every engineer is paralyzed with fear at committing substantial changes because the application is one Jenga block from collapse.
We can’t maintain that breakneck pace anymore because it would actually break our necks and probably our product. No one has bothered to take out the technical trash. We have a frightful amount of manual QA because no one trusts our code. We’re moving slowly and still breaking things, as last night’s outage proved.
“I don’t want complaints; I want solutions,” you say. Great, I agree completely. Here come the numbers.
Our average Commit-to-Deploy time (CDT) is around 30 minutes. That’s the time it takes for builds to get out of code review and ready for deploy. Here’s a breakdown of those 30 minutes:
- 15 min to build & run automated tests
- 5 min to build & deploy to staging
- 10 min for QA & spot checking
A 30-min CDT doesn’t seem that long, but it has rippling effects:
30 minutes is long enough that engineers will try to optimize for efficiency. Committing a one-liner that still takes 30 minutes to get to production isn’t an optimal use of time, so they’ll squeeze more code into one pull request.
Longer pull requests take more time to review. And reviewing code already requires a considerable amount of time. Attention fades when a reviewer is asked to look at more than 400 lines of code. So, you wind up with worse code reviews for larger amounts of code.
Lengthy code reviews increase the likelihood that changes will be requested. Expecting these changes, the PR’s author will have to maintain familiarity with more code. Even if they manage to get work done on another feature, they still incur the costs of switching contexts.
These same costs are incurred when engineers enter the deploy queue. For 30 minutes, they can do nothing seriously but monitor the status of their build and subsequent deploy. This is enormously wasteful, in terms of both time and money.
Assuming an average engineer is paid $50/hr and deploys twice per day, that’s $13,000/yr. For our team of 8 engineers, that’s $104,000/yr. So, our infrastructure is actually costing us an extra engineer’s salary. Awesome.
Using a CI/CD tool like CircleCI (BLATANT PRODUCT PLACEMENT!!) would save us both time and money. $9,000/yr buys us 16 containers, on which we could run 4 concurrent builds 4 times as quickly. This would work great for our team of 8, since we’re rarely trying to deploy more than 4 features at a time.
Let’s say this parallelization cuts our automated testing time in half: about 8 minutes. Let’s also say that our engineers are merging smaller, more reviewable changes, because they’re not gaming the system anymore. These smaller PRs mean QA is faster, so let’s shorten that to 5 minutes. If we’re building/deploying for staging with CircleCI, then that shaves off about 3 more minutes.
All of that brings us down to a CDT of 15 minutes. We spent $9,000 and got back $52,000, for a net gain of $43,000.
We just got time and money back by implementing a philosophy and tool.
I get that this won’t happen overnight; it will happen overweek, probably even overmonth. But it needs to happen.
Writing good tests is hard, and many of our engineers don’t even think writing average tests is required for “getting shit done.” Those engineers are absolutely right: it’s not strictly necessary to write tests if you don’t mind having shit in your codebase.
Look, humans aren’t great at thinking long-term; it’s just not how we’re wired. So, it’s not surprising that we favor short-term wins and churning out features as fast as we can.
But in order to maintain this pace, we need to slow down and fix our process. And I realize that sounds counter-intuitive, but it’s necessary if we ever want to get out of technical debt and back into what we all want to be doing: building software that customers love.
Let’s make this happen.