Configuration files can take some time to set up, but after that initial push they are easy to forget about. “If it’s not broken, don’t fix it” is a common approach that many developers take with their configuration files. But when it comes to your continuous integration pipelines, small changes can have huge benefits. At CircleCI, we are pushing out new features and updates faster than ever before, and regularly reviewing and updating your config can lead to significant reductions in workflow duration.
To help customers that have invested in our support plans take full advantage of CircleCI, our DevOps Customer Engineering (DCE) team offers reviews of configuration files on a regular basis. The DCE team has reviewed hundreds of configuration files and has created a process that can be used with any project. In this article, we will be going through the DCE’s review process to enable you to conduct your own config review. You could be the one that decreases the feedback loop for your pipelines. Imagine your team being able to push code and receive feedback in the time it takes to refill their coffee, rather than the time it takes to get lunch.
What is a configuration review?
A configuration review is more than looking over the YAML file. It should be treated as a key process, be planned out, and be documented. We recommend choosing a project and performing a review at least once a year, if not every six months, then capturing the results in a document accessible to the rest of the team. Documenting the baseline metrics, what was done during the review, and the results is arguably just as valuable as the review itself. With that information, your team can build up a library of CI knowledge that is specific to your use case and can be used for future reviews or new projects.
The primary goal of a configuration review is to analyze your project’s configuration file to ensure pipelines are running optimally when compared to your metric goals. Our Insights dashboard is a great way to track metrics during this process. If your team uses another metric application, you can use our Insights API to pull data into it. Performing a configuration review for your team is also a great way to learn how CircleCI works. We recommend using this time to document how features work with your use case and how they’ve been implemented for your workflows.
Preparing for a configuration review
Before you dive into reviewing your configuration, there are a few important steps that will help ensure your review is effective. In the preparation stage, you’ll want to choose a project to review, determine your goals for the review, and establish some baseline metrics you can use to measure your progress toward your goals.
Choosing a project and workflow to review
Most teams have more than one project, and reviewing multiple project configurations can be overwhelming. When DCEs work with a customer to review their configuration file, we work together to decide on a particular project to focus on. The top three reasons for choosing a project are:
- High credit consumption
- Long workflow durations
- Overly complex configurations
There are plenty of other reasons for choosing a project, too. Maybe it is a proof of concept for other projects, one that hasn’t been touched in a while, or one with a concerning metric.
Deciding on a goal
After choosing a project, you’ll want to decide on a goal for the review. Are you looking to increase developer productivity by reducing workflow duration? Does credit consumption need to be optimized? Maybe you are looking to implement new CircleCI features since the last time the configuration was reviewed.
Whatever goal your team chooses, write it down so that anyone doing a review in the future can understand which approach was used for the review.
Using Insights to gather baseline metrics
Now that you’ve set a goal, how will you know if your review process was successful?
CircleCI is one of the few CI tools that offers Insights into your team’s metrics. Using the Insights tool, you’ll want to gather the project’s and/or workflow’s four key metrics:
- Duration
- Throughput
- Success rate
- Mean time to recovery
If you’d like more information about why we consider these the four key metrics, we recommend reading through our State of Software Delivery Report (2022) where we analyze them in more detail.
These metrics might not all be necessary to track your goal’s success, but it’s helpful to note them down to set a baseline. After changes to the configuration file have been implemented, we’ll take additional metric snapshots and compare them to this one to see how the changes have worked out.
This is also a good time to determine which metrics are important to this project. Each project is different, so the metrics used to determine each project’s success will be different too. If you are optimizing for credit consumption, then duration is a great metric to focus on since that directly affects credits. If your goal is overall efficiency, along with the duration you’ll want to pay attention to resource usage on jobs that use a Docker executor.
Beyond the four key metrics, there may be additional metrics that are important for individual workflows. Many of them, like flaky tests and credit usage, are included in Insights. If there are metrics that you’d be interested in seeing added to the Insights tool, let us know about them.
Reviewing your CircleCI configuration file
Once you’ve determined your goals and recorded your baseline metrics, you’re ready to start the meat of the review. If you are using your review as an opportunity to learn more about your pipelines and/or CircleCI, it’s recommended you read through the next section to get more familiar with your team’s use case. If you are comfortable with how your team’s pipelines work, feel free to skip ahead to diving into the config file.
Getting to know your workflows
If you are new to CircleCI and/or your team’s workflows, we recommend you start reviewing the project from the Insights dashboard. Having a general understanding of what is expected in the workflow and the outcome of them is helpful when reviewing the config. Using the Insights tool, you can review metrics for specific workflows and jobs within the project. The Insights dashboard displays up to the last 100 workflow runs and you can investigate any outlying runs. It’s also recommended to click into the last 10 or so workflow runs to ensure that jobs are running in a consistent manner. Reviewing the steps in each job, their outputs, and their individual metrics can also be helpful in determining any inconsistencies.
Diving into the configuration file
Opening the configuration file in the UI will give you access to any notification banners or recommendations that CircleCI automatically provides. Don’t feel like you need to tackle these issues first. Note them down for later when we go through the configuration file in depth. Throughout the review, take notes about changes related to your goals as well as any questions that arise and their answers. This is also a good time to note any small issues that can be addressed, like out-of-date comments or inconsistent naming conventions.
At this point feel free to switch to your favorite code editor if you feel more comfortable with it. You spent time setting up those colors for a reason. Over the course of the review, we’ll be going between the workflow/job UI and the config file so it’s handy to have both open. If you haven’t already, don’t forget to open the config file in the UI at least once to see any important notifications.
Configuration file layout
CircleCI’s configuration file doesn’t require a specific order for the configuration file version, setup, orbs, jobs, or workflows. It is up to your team how you’d like to organize them. However, the following format has been successful for many teams:
- Config file version (currently 2.1)
- Setup, if your team uses dynamic config
- Customized logic such as:
- Orbs
- Commands
- Parameters
- YAML anchors
- Jobs
- Workflows
- Commit triggered workflows
- Workflows set up to trigger via scheduled pipelines
Customized logic, executors, and orbs
The configuration file version and using the setup boolean are pretty straightforward so let’s jump into reviewing the various customization options that CircleCI provides, starting with customized logic.
Customized Logic
Customized logic includes commands, parameters, executors, and YAML anchors. You’ll most likely need to keep all the parameters you currently use, but it’s a good idea to review them and see if another feature, like contexts, would be a better fit. The commands, executors, and YAML anchors we can do a bit more with. It’s really easy to write quick customization in the configuration file when you need one, but after you’ve added a few it can start to reduce the configuration file’s readability. That’s where orbs can come in to help.
Orbs
If your team doesn’t use orbs already, they can be very handy. Essentially they are packages of YAML logic that can be accessed with a command instead of writing out the full logic in the configuration file. There are many certified and third party orbs already available, and it’s a good idea to search through the orb registry to see what would be helpful for your tech stack.
If your team would rather not use a public orb, CircleCI also offers private orbs which restrict an orb to only be available to your organization. Orbs can be useful for extracting lengthy code, code used more than once, or code used in multiple projects from the configuration file. As you go through your configuration file, consider what you could transfer to an orb.
If your team is already utilizing orbs, you’ll want to check if there are any newer versions of the orbs you are using. There may be additional commands, jobs, or other features available that can be found on the orb’s page in the orb registry.
Executors
CircleCI offers a wide variety of execution environment options to choose from so reviewing the ones currently being used can be a great way to reach your review’s goals.
As a first step it’s a good idea to check if the execution environment has had an update. Assuming your team is using a convenience Docker image, head over to the image repository to see what updates are available. If your team isn’t using one of these images, why not check them out? They are well cached which reduces spin-up times. If you are using another execution environment, you can find a list of available options in the configuration reference documentation and check for any updates.
Along with reviewing which executor should be used, you can confirm if the resource class currently being used is the right fit for the job. When using the Docker executor, CircleCI offers the ability to monitor resource consumption over the life of the job, which is handy for this stage of the review. You can also see the job resource usage history over multiple runs on the Jobs tab in Insights. We’re looking to add the same features for other execution environments in the future, but for now, if you are using an executor other than Docker, you’ll need to add resource-logging into the job yourself.
By default, new jobs will use a larger resource class, so for simple jobs like linting or an API call, consider lowering the resource class size. We recommend always setting your own resource class in the configuration file rather than using the default.
Jobs and features
Every job is unique, so it can be difficult to provide blanket recommendations on how to optimize at the job level. The good news is that CircleCI offers quite a bit of visibility into what is happening during a job. Which job you want to start looking at first is up to you, but eventually, you’ll want to look at each job in the UI to confirm that it’s working as intended and to see if you can optimize it according to your configuration review goals.
Start by opening the job in CircleCI and reviewing each step one by one. It can be helpful to review multiple runs of the job to confirm if something is an outlier or not. Some common issues that can be caught with this method are:
- Caching or workspaces not working as intended
- Test splitting reverting to the default of splitting by name
- Tech stack warnings
If reducing duration is one of your goals, it’s a good idea to look into introducing or increasing a job’s parallelism. Used with test splitting, parallelism can drastically reduce duration, depending on how long it takes to set the tests up. If your goal is credit optimization, consider if the number of containers being used could be reduced.
As you scan through your jobs you may find that the same tests are being run in different execution environments. To help with configuration readability, consider if matrix jobs could be utilized. You may also find that logic is being repeated in different jobs. This is another place where orbs can help by extracting that logic to make it not only available to this project but also across your entire organization.
Depending on when you last did a configuration review, there may have been some new features released that your team could take advantage of. Our public roadmap and our changelog are great resources to see what has come out recently. While not all features on CircleCI are available in the configuration file, another resource is the configuration file reference and where you can read about any feature that you are unfamiliar with.
Workflows
Similar to jobs, workflows tend to be unique. While there are standard workflows like build-test-deploy, every organization has their own spin on how to set one up. Just like with jobs, start by opening the workflow in CircleCI.
There are a few general tips to keep in mind as you review the workflow, the primary one being whether the job order makes sense for your current needs. Maybe you could provide feedback to your developers by failing faster or placing your shorter tests before longer ones. Some of your jobs might be running in sequence when they could be run in parallel. Creating the right fit for your team’s needs might take some experimentation, and that’s okay. The Insights dashboard offers many features and metrics that can help you decide which workflow setup is right for you.
Lastly, if you are using any filtering or logic statements, you’ll want to confirm that they are working as intended. If you aren’t using those features, consider adding them to your workflows to help ensure jobs run only when they need to, like a deploy job running only on the default branch. Check them out and see if they are right for your team.
Post review
Now that the review is done there are a few more steps to take but the hard part is done. Great job!
Credit where credit is due
As you were going through the configuration review, someone on your team may have implemented something that saved everyone time. Maybe there is some logic that is useful in another project that you weren’t aware of. If you do find something cool, give the person who implemented credit for it. Everyone deserves recognition for their work, including you for doing the review.
Implementing changes
When implementing changes it’s important to identify the expected result. If your team has decided to increase parallelism on a job, it might be a good idea to test that change by itself so the metrics aren’t obscured by other changes. If you find that there are multiple changes that have a smaller expected result, feel free to group those together when testing. Creating a test branch for these changes will allow you to quickly determine how a change will affect your metrics in the short term and allow you to test various degrees of change, such as different levels of parallelism.
Comparing metrics
Whichever way you choose to implement changes, it’s a great idea to use Insights to take metrics before and at regular intervals after. Once changes are made, it’s recommended to snapshot the project’s metrics at progressively increasing intervals. Not only will this confirm or disprove the expected results, but it will also allow you to monitor for any metric fluctuations.
Still not convinced about how Insights can help you review and optimize your configuration file? Check out how our Insights team used Insights to perform their own configuration review.
Conclusion
Throughout this process, you’ve learned how to use Insights to set goals for a project, review a CircleCI configuration file, and implement changes to reach those metric goals. We’ve discovered that CircleCI offers everything you need, from Insights to UI banner recommendations, to help you on your configuration review journey. I hope the value of taking a dive into your team’s configuration files has been laid out for you here.
If you made it to this point, congratulations! You may have noted that this process can be quite involved and said to yourself, “I don’t have the time for this, but I see the benefit.” The great news is that those DevOps Customer Engineers we talked about earlier can conduct a configuration review for your team. Our DCEs have done numerous config reviews and have compiled data on best practices for a wide variety of use cases. Along with configuration reviews, we offer many Value Added Services that your team can take advantage of, including help with project migration to CircleCI and feature training. If your team would like to meet with our DCEs for a configuration review, feel free to reach out to our Sales Team to upgrade your support plan.
