Instead of manually creating an individual CircleCI configuration per project, you might prefer in some cases to generate these configurations dynamically, depending on specific pipeline values or file paths.
CircleCI’s dynamic configuration feature allows you to:
- Execute conditional workflows/commands.
- Pass pipeline parameter values and/or generate additional configuration.
- Trigger separate
config.ymlconfigurations, which exist outside the default parent
This becomes particularly helpful in cases where your team is using a monorepo, or a single repository, as opposed to using multiple repositories to store your code.
For instance, you may want to use dynamic configuration to trigger certain workflows based on which files or subdirectories have changed in your repository. Otherwise, all of your microservices/sub-projects would go through the entirety of your build, test, and deployment processes when any single update is introduced. This scenario is also referred to as path filtering.
Another scenario in which you could use dynamic configuration is when your project consists of multiple modules, where each module requires a separate job. These jobs could be stored in multiple files as fragments of a configuration. Instead of using one full
.circleci/config.yml file, you could use dynamic configuration to combine these individual fragments into a full configuration when a pipeline is triggered. This scenario can also be referred to as config splitting.
Getting started with dynamic config in CircleCI
To get started with Dynamic Config in CircleCI:
- In the Projects dashboard from the CircleCI web application, select your project.
- Click the Project Settings button in the upper-right corner.
- On the left-hand panel, select Advanced.
Scroll to the Enable dynamic config using setup workflows setting, and toggle it to the “on” position, as shown below:
- While the steps above will make the feature available, your static
config.ymlwill continue to work as normal. This feature will not be used until you add the
setupkey with a value of
config.yml. Adding the key
setup: trueto the top level of your parent configuration file (in the
.circleci/directory) will designate that
config.ymlas a setup configuration.
- At the end of the
continuejob from the
continuationorb must be called. NOTE: This does not apply if you want to conditionally execute workflows or steps based on updates to specified files (“path filtering”), as described in the Configuration Cookbook example.
How dynamic config works
CircleCI’s dynamic configuration feature uses setup workflows. A setup workflow can contain jobs that compute pipeline parameters, which can then be passed into an additional configuration that potentially exists in other directories. A setup workflow can also be used to generate new configurations via pre-existing scripts. In any case, the setup workflow continues the pipeline on to the desired next configuration.
Behind the scenes, the continuation configuration is implemented as a call to a public pipeline continuation API. This API accepts a continuation key, which is a secret, unique-per-pipeline key that is automatically injected into the environment of jobs executed as part of the initial setup workflow. The API also accepts a configuration string, as well as a set of pipeline parameters.
- The setup phase requires
version: 2.1or higher.
- A pipeline can only be continued once (that is, a pipeline cannot be continued with another setup configuration).
- A pipeline can only be continued within six hours of its creation.
- The setup configuration can only include one workflow.
- Pipeline parameters submitted at continuation time cannot overlap with pipeline parameters submitted at trigger (setup) time.
- Pipeline parameters declared in the setup configuration must also be declared in the continuation configuration. These parameters can be used at continuation time.
For a basic example on how to use setup workflows for dynamic configuration generation, see the Configuration Cookbook.
Dynamic configuration FAQs
Q: I thought pipeline parameters could only be used with the API?
A: Previously, this was true. With the dynamic configuration feature, you can now set pipeline parameters dynamically, before the pipeline is executed (triggered from either the API or a webhook—a push event to your VCS).
Q: Can I use a custom executor?
A: Custom executors can be used, but require certain dependencies to be installed for the continuation step to work (currently:
The continuation orb
Q: What is the
continuation orb assists you in managing the pipeline continuation process. The
continuation orb wraps an API call to
continuePipeline. Refer to the
continuation orb documentation for more information.
Q: Is it possible to not use the continuation orb?
A: If you have special requirements not covered by the continuation orb, you can implement the same functionality in other ways. Refer to the orb source code to learn how the continuation functionality is implemented with the orb.
What to read next
- Cookbook examples
Take the dynamic configuration course with CircleCI Academy to learn more.
Help make this document better
This guide, as well as the rest of our docs, are open source and available on GitHub. We welcome your contributions.
- Suggest an edit to this page (please read the contributing guide first).
- To report a problem in the documentation, or to submit feedback and comments, please open an issue on GitHub.
- CircleCI is always seeking ways to improve your experience with our platform. If you would like to share feedback, please join our research community.
Our support engineers are available to help with service issues, billing, or account related questions, and can help troubleshoot build configurations. Contact our support engineers by opening a ticket.
You can also visit our support site to find support articles, community forums, and training resources.
CircleCI Documentation by CircleCI is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.