At Circle, our traditional approach to Kubernetes (k8s) deployments likely looks familiar to many of you: Run the workflow, create the image, build the Helm chart and deliver it to k8s. At that point, k8s takes over with its rolling update.
This method gets the job done, but we knew it wasn’t ideal. Limited support for canary releases and the need for time-consuming error monitoring and manual rollbacks added friction and risk to our release processes.
We knew we wanted to introduce progressive deployments—the question was how do we get there, with the least disruption to our teams and end users. In this post, we’ll describe our process and reasoning behind adopting Argo Rollouts internally, and what CircleCI customers can expect to see in the near future as we introduce more advanced and intuitive deploy and release features across our platform.
Evaluating the progressive delivery landscape
Our journey towards progressive delivery started with the acquisition of Vamp in May of 2021. That team sharpened our deployment tooling expertise and gave us greater insight into the needs of developers working with cutting-edge release strategies. As we integrated the offering, we observed that the broader software community had begun to embrace progressive delivery along with a number of associated open source tools.
Looking at our options, we had a choice: build our own controller for progressive delivery or explore existing open-source tools that aligned with our roadmap.
Seeing the movement toward these new tools, we took a fresh look at the available options in the OSS landscape. We recognized that our experience gave us a unique opportunity to build value-adding differentiators rather than replicate existing tools. Ultimately, we found that Argo Rollouts offered the best functionality to both meet our own deployment needs and to serve as an important piece of the deploy and release toolset we are building for customers.
Why Argo Rollouts
A few tools offered comparable features. What set Rollouts apart was its canary strategy: It provides basic traffic routing without the need to introduce a service mesh to our cluster. Yes, the traffic routing percentages would be coarse, but that wasn’t a deal breaker. We could always explore introducing a service mesh later for more fine-grained control of traffic weights.
There were other features we liked:
- Flexible and powerful Analysis Runs (automated metrics assessment)
- The ability to run experiments without exposing them to live traffic
- The project is open source
Rollouts’ automated experimentation and canary analysis features simplify the process of releasing and validating changes, eliminating manual toil and increasing developer confidence. And the fact that Argo is an open source project gives us a unique opportunity to share our knowledge and experience. We’re excited to begin contributing to making it better for the entire community.
Progressive deployments with Argo Rollouts
Our primary goal was to achieve safer deployments. Adopting progressive deployment with the canary strategy, we aimed to make soaking changes a part of our release process. Under our old process, canarying changes was a time-consuming manual process that disincentivized following the safest route to production. Now, once onboarded with Rollouts, teams face more friction to avoid canarying their changes.
Aligning our incentives with best practices allows us to ensure each release is rolled out in the safest way possible, without adding to our engineers’ workloads. One of the valuable features of Argo Rollouts is the ability to release changes in steps. So we can set weight to 10% of traffic, let that soak for 15 mins, then move to 50%, wait 20 mins, then 100%. While this is happening we have some analysis runs in the background, checking for container restarts or an increase in logged errors. If we cross our redline, the release is canceled, we go back to our stable pods, and the team is notified via Slack.
One of the most common concerns we heard from our teams as we introduced Rollouts was that the new process could prolong deployments. Yet it’s important to distinguish between deployments and releases. Deploy is how fast we get our changes out to the production environment, and we still aim to deploy changes as quickly as possible. But at that point release takes over - how fast we make the new version available to customers. And we’re not convinced going to 100% on release as fast as possible is the right approach. Regardless of the number of tests and environments in place - whether we admit it or not - we all inevitably test in production.
Results so far
Onboarding teams to Argo Rollouts has been pretty straightforward for us. Most of our deployments go through shared tooling, which simplifies adoption. We were able to add Rollouts to our common chart. Following a checklist, teams can onboard their services to Argo Rollouts in about an hour—including pipeline run time—by modifying their values.yaml.
To date, over 80% of our deployments are using rollouts, and the results have been overwhelmingly positive:
What’s next?
At Circle, we’re big believers in dogfooding. Some of our best work and features have emerged from using our own tools, solving our own problems. When we decided to adopt Argo Rollouts internally, we knew there would be friction experienced by our developers. They would need to go outside the standard CircleCI UI and instead use the Rollouts dashboard to manage and monitor their releases.
Once we knew Rollouts was a great fit for us on our Engineering team, we began work on extending the CircleCI UI to include deploy and release features, first for our engineers and ultimately for all users. The goal is to give our customers maximum visibility and control by making the CircleCI UI the only pane of glass you need to manage your releases. With the success our team has seen integrating this tool, we know this is also the right path for all our customers.
Stay tuned for more updates on our integration with Argo Rollouts. Moving forward, we will share more details, including some lessons learned during our onboarding process and experiences working with Analysis Templates. We hope these insights will be valuable and encourage you to explore this space further.
Get started with progressive deployments on CircleCI with Argo Rollouts
If you’ve been looking to get started with progressive deployments, by all means try Argo Rollouts. We like it, and we’re confident you will too. You can incorporate Argo into your pipelines today by following the steps in our progressive delivery tutorial. In the near future, we’ll make this process even easier by building deploy and release features directly into CircleCI so you can manage releases natively within your pipelines.
CircleCI is committed to helping our customers validate changes wherever they occur so they can ship value faster and more confidently. Sign up for a free account and get up to 6,000 build minutes per month to begin your journey toward more stable and consistent releases.
