Instead of manually creating an individual CircleCI configuration per project, you might prefer in some cases to generate these configurations dynamically, depending on specific Pipeline values or file paths.
CircleCI’s dynamic configuration feature allows you to:
Execute conditional workflows/commands.
Pass pipeline parameter values and/or generate additional configuration.
config.ymlconfigurations, which exist outside the default parent
This becomes particularly helpful in cases where your team is using a monorepo, or a single repository, as opposed to using multiple repositories to store your code.
For instance, you may want to use dynamic configuration to trigger certain workflows based on which files or subdirectories have changed in your repository. Otherwise, all of your microservices/sub-projects would go through the entirety of your build, test, and deployment processes when any single update is introduced. This scenario is also referred to as path filtering.
Another scenario in which you could use dynamic configuration is when your project consists of multiple modules, where each module requires a separate job. These jobs could be stored in multiple files as fragments of a configuration. Instead of using one full
.circleci/config.yml file, you could use dynamic configuration to combine these individual fragments into a full configuration when a pipeline is triggered. This scenario can also be referred to as config splitting.
Getting started with dynamic config in CircleCI
To get started with Dynamic configuration in CircleCI:
In the Projects dashboard from the CircleCI web application, select your project.
Click the Project Settings button in the upper-right corner.
On the left-hand panel, select Advanced.
Scroll to the Enable dynamic config using setup workflows setting, and toggle it to the "on" position, as shown below:
While the steps above will make the feature available, your static
config.ymlwill continue to work as normal. This feature will not be used until you add the
setupkey with a value of
Adding the key
setup: trueto the top level of your parent configuration file (in the
.circleci/directory) will designate that
config.ymlas a setup configuration.
At the end of the
continuejob from the
continuationorb must be called. This does not apply if you want to conditionally execute workflows or steps based on updates to specified files ("path filtering"), as described in the Using dynamic configuration how-to guide.
How dynamic config works
CircleCI’s dynamic configuration feature uses setup workflows. A setup workflow can contain jobs that compute pipeline parameters, which can then be passed into an additional configuration that potentially exists in other directories. A setup workflow can also be used to generate new configurations via pre-existing scripts. In any case, the setup workflow continues the pipeline on to the desired next configuration.
Behind the scenes, the continuation configuration is implemented as a call to a public pipeline continuation API. This API accepts a continuation key, which is a secret, unique-per-pipeline key that is automatically injected into the environment of jobs executed as part of the initial setup workflow. The API also accepts a configuration string, as well as a set of pipeline parameters.
For a basic example on how to use setup workflows for dynamic configuration generation, see the Using dynamic configuration how-to guide.
I thought pipeline parameters could only be used with the API?
Previously, this was true. With the dynamic configuration feature, you can now set pipeline parameters dynamically, before the pipeline is executed (triggered from either the API or a webhook—a push event to your VCS).
Can I use a custom executor?
Custom executors can be used, but require certain dependencies to be installed for the continuation step to work (currently:
What is the continuation orb?
Is it possible to not use the continuation orb?
If you have special requirements not covered by the continuation orb, you can implement the same functionality in other ways. Refer to the orb source code to learn how the continuation functionality is implemented with the orb.