This post, written by CircleCI Technical Content Marketing Manager Ron Powell, originally appeared on The New Stack here.
No one likes giving their weekends up to fix release issues.
Developers and operations teams are traditionally hesitant to make changes or deploy applications on a Friday, in case something goes wrong and they have to spend their weekend making emergency fixes. Or worse, trying to roll back changes that were made.
However, with a strong set of practices and a reliable deployment pipeline, there should be no reason why a deployment cannot happen anytime — even on a Friday afternoon.
In this post, we’ll cover some of the steps DevOps teams should take to ensure that they can safely and reliably deploy almost any change and not lose their weekends.
What do I need to deploy on a Friday?
To ensure that you can deploy on a Friday, or at any time, you must be sure that your application will — at a minimum — have no critical bugs that require emergency fixes or rollbacks.
To get this sort of certainty, you need a suite of tests that you can trust to cover as many use cases as possible. You also need a way to ensure that these tests are run and passed before you deploy your application to production. But how do you even start to develop tests and add them to your development pipeline so that you can get this confidence? It’s always best to begin small and iterate. Develop a simple plan, like the one below, to increase your testing with the resources you have.
What do I need to deploy at will?
Deploying applications at any time is a lofty goal. So, what do you really need to deploy applications at will? Confidence.
Confidence is what allows you to deploy applications at any time without fear of losing a weekend trying to fix critical bugs that made it into production. Which leads us to the reason why most developers and operations teams lack confidence in their deployment: insufficient testing to cover the whole application.
Without rigorous testing across most — if not all — of the applications and environments you are deploying to, bugs will almost certainly make it into production. Sometimes, you’ll get lucky and those bugs won’t be critical. Other times, you will need to make emergency fixes or roll back changes — which leads to a lack of confidence in your deployments.
You need that confidence in your pipeline, as well as the changes going through it, to ensure that your application is always ready to deploy. You also need a way to automate as much of this process as possible, to ensure that your deployments require no extra effort.
The testing journey: starting out
Start by looking at your application’s critical components — parts of the application users rely on the most. Develop some basic tests to check that this functionality works. You don’t need to cater for every single possible combination right away, just come up with tests that check the “happy path” — the one you expect users to see most of the time.
If you have time and know what your users sometimes do outside of this path, add a few test cases to check those as well.
Adding more tests
The next step for getting more confident in your deployments is diligently adding both unit tests and integration tests any time you fix a bug or add a new feature.
Fully adopting the Test Driven Development (TDD) practice helps, as this ensures that any new or changed code will always have tests. Even if you don’t fully adopt the TDD approach, writing a few tests each time you fix a bug — to prove the solution is viable or check if the new feature meets requirements — is a good start.
Once you start getting into the habit of introducing tests and aligning them with your process, incorporate those tests into your continuous integration pipeline. Tests incorporated as part of your continuous integration (CI) build process will rerun automatically, before a new build is deployed.
With all these tests running across all parts of your application, your confidence that no bugs have made it into production will increase, since you have automatically tested most of your application at this point.
Fixing the inevitable
Once you start writing and automating a large amount of test cases, you will inevitably begin catching failures as the change enters the build pipeline. This is where a quick, standard process for flagging test failures back to the committers of a change should be developed.
You want the build to happen and be tested as quickly as possible, so that the developers are notified when changes pass or — most importantly — fail any tests, as quickly as possible. This promotes your CI pipeline as the solution to the question, “How do we ensure our code can be deployed at will?”
Adding tests to your CI pipeline
Now that you have a general game plan for adding tests to our codebase, how do you implement this plan?
Most of the time, you want your testing to line up with your environments in order to segregate the tests. June Jung has published a great, in-depth post about test segregation. To sum it up, you want to establish three tiers:
- Unit tests: small, cheap, fast tests that cover basic code functions without dependencies, to be run in development when the code is checked in.
- Integration tests: larger, cross-unit tests that cover how services integrate, to be run through the early test environments.
- User Interface (UI) tests: automated, interface-based testing for user workflows, to be run just before or after deployment to production.
One other critical thing to think about when adding tests to your pipeline: do you have enough code coverage? When you are starting out adding tests to an existing application, your coverage is naturally low. There is, however, no single definitive percentage of code coverage to aim for.
The best way to proceed is to evaluate your coverage, then aim to gradually increase it over time. It is important not to go “all-out” and aim for 100% coverage, as you could cover 100% with terrible tests that aren’t thorough enough and let bugs slip through.
Circling back to our game plan, start with the critical parts of the application to ensure they are well covered by realistic tests. There is no real way to know for sure if you have full coverage. This is about increasing confidence, by testing as much as you can while developing your code.
As you think about scenarios that might break your code more and more, your tests will become stronger and more complete — allowing you to catch those critical issues. For a deeper look, June has another great post on mocking, stubbing, and contract testing.
Achieving deployment freedom
Good test coverage of your application, built over time and focused on the critical parts first, is key to developing confidence in your deployments. By integrating the suite of tests into your CI pipeline and ensuring they are run every time code is committed, you will quickly gain the confidence to deploy your application at any time — even on a Friday afternoon.
As your code coverage becomes more complete and you become even more confident, you can also add continuous delivery (CD) to your pipeline. With a full CI/CD pipeline running, every build will be automatically tested; and if it passes all the tests, it will also be automatically deployed in one go.
CI adoption also changes developer team culture. When it comes to developer team success, finding the right DevOps metrics to measure is crucial. Learn how to measure DevOps success with four key benchmarks for your engineering teams in the 2020 State of Software Delivery: Data-Backed Benchmarks for Engineering Teams.