Dynamic configuration overview

Cloud

Server v4+

The work is determined by values passed in at runtime as pipeline parameters.
The configuration file is dynamically generated by the pipeline using a template rather than being static YAML
A project’s configuration is pieced together dynamically depending on specific Pipeline values or file paths.
Using conditional logic and parameterization to determine the work to be done.
Triggering pipelines defined by configurations that exist outside the default parent .circleci/ directory.

Quickstart

Enable dynamic configuration

Projects created after December 1st 2023 have dynamic config enabled by default. If this applies to your project, you can skip this step.

To enable dynamic configuration in CircleCI:

In the CircleCI web app, select Projects in the sidebar
Select the next to your project and select Project Settings
From the sidebar, select Advanced.
Scroll to the Enable dynamic config using setup workflows setting, and toggle it to the "on" position, as shown below:

These steps make dynamic configuration features available for your project, but any static config.yml you have previously set up will continue to work as normal.

Use dynamic configuration

Add the key setup: true to the top level of your parent configuration file to designate that config.yml as a setup configuration.

version: 2.1

setup: true

For some basic examples of using dynamic configuration, see the Using dynamic configuration how-to guide.

Config continuation constraints

Some constraints on continuing pipelines from a setup: true configuration are as follows:

The setup configuration must be version: 2.1.
The setup configuration must only allow one workflow to run in each pipeline. For example:
- Have only one workflow configured.
- Use logic conditions to ensure only one workflow is run for each pipeline.
A pipeline can only be continued once. A pipeline cannot be continued with another setup configuration. Workflow reruns will fail as part of this restriction. Rather than rerun a setup workflow you can trigger a new pipeline.
A pipeline can only be continued within six hours of its creation.
Pipeline parameters submitted at continuation time cannot overlap with pipeline parameters submitted at trigger (setup) time.
Pipeline parameters declared in the setup configuration must also be declared in the continuation configuration. These parameters can be used at continuation time. For more information, see the Pipeline values and parameters page.

How dynamic config works

CircleCI’s dynamic configuration is made up of the following resources, which can be used together to fit a variety of use cases:

The setup config field allows you to designate a configuration file as dynamic, allowing the use of dynamic configuration features. When a configuration is dynamic (setup: true) the CIRCLE_CONTINUATION_KEY environment variable is available, which is a secret, unique-per-pipeline key that is automatically injected into the build environment for jobs that run in a setup configuration.

The CIRCLE_CONTINUATION_KEY environment variable is required to authorize the continuation of one configuration to another via the continue a pipeline API v2 endpoint. The API also accepts a configuration string, as well as a set of pipeline parameters.

A setup configuration:
- Continues your pipeline on to the desired next configuration.
- Can contain jobs that compute pipeline parameters, which can then be passed into an additional configuration that potentially exists in other directories.
- Can be used to generate new configurations via pre-existing scripts.
Use the API v2 continue a pipeline endpoint to continue a pipeline from one configuration to another. This endpoint can be used directly or via one of the dynamic configuration orbs described next.
Use the path filtering orb to map file paths, pipelines parameters, and configuration files to determine the work that should be done when a change happens in a specific location in a repository.
Use the continuation orb to continue one configuration to another and set up any required pipeline parameters.

Dynamic configuration examples

Some examples of using dynamic configuration in CircleCI include:

Incorporate logic to check for file changes or parameter values so that only necessary work is done in each pipeline.
Share common configuration across projects by generating project-specific sections dynamically based on parameters.
Continue a pipeline, or generate and run a configuration based on where code has changed.
Split up a configuration file into reusable fragments that can be dynamically assembled.
Generate full CircleCI configurations from small company-specifiic configurations. For example, if you have many repositories with similar CI/CD requirements, you can generate a full CircleCI YAML configuration dynamically from small template configuration written in any configuration language.
When working with a language or framework that has its own configuration language, you can generate CircleCI config dynamically from existing configuration files.

Use path filtering for monorepos

Set up path filtering to set pipeline parameters and continuation configuration sources based on the location of changes in your repository.

In a monorepo (single repository) setup you can use path filtering to avoid all of your microservices/sub-projects being put through the entirety of your build, test, and deployment processes when any single update is introduced.

For example, you may have multiple projects within your repository (project_1, project_2, project_3). You can use path filtering to only run tests and deployment for project_1 if change happened in the project_1 file set. The path filtering orb is available to help simplify your configuration further. For a full example of path filtering, see the Dynamic configuration how-to guide.

Use config splitting for microservices

For projects that consist of multiple modules, you may require a separate job/workflow to run for each module when changes are introduced. You can split your configuration up and store the config for each module separately within the module’s directory, keeping related code and config together. Module jobs/workflows can be stored in multiple files as fragments of a configuration. Instead of using one full .circleci/config.yml file, you can use dynamic configuration to combine these individual fragments into a full configuration when a pipeline is triggered. This scenario can also be referred to as config splitting.

For an example of config splitting, see the Dynamic configuration how-to guide.

Using dynamic configuration on CircleCI server

For more information on managing orbs in a server installation see the Managing orbs page.

If you are using CircleCI server there are a few extra steps you will need to take to use the dynamic configuration orbs.

You will need to import the path filtering and continuation orbs into your server installation using the following commands:

These examples show importing the most recent versions of the orbs. You can also specify a SemVer version constraint to import a specific version, for example, circleci/continuation@1.0.0.
```
circleci admin import-orb circleci/path-filtering --host <your-server-installation-domain> --token <your-api-token>
```
```
circleci admin import-orb circleci/continuation --host <your-server-installation-domain> --token <your-api-token>
```

When using the path filtering orb on server you will need to add your CircleCI server domain to the path-filtering/filter job in your workflow, for example:

workflows:
  setup-workflow:
    jobs:
      - path-filtering/filter:
          mapping: |
            packages/api/.* api true
            packages/auth/.* auth true
            packages/e2e/.* e2e true
            packages/unit/.* unit true
          # # Optional, defaults to main:
          base-revision: main
          circleci_domain: support-server-4.eks-dev.sphereci.com

FAQs

I thought pipeline parameters could only be used with the API?

Previously, this was true. With the dynamic configuration feature, you can now set pipeline parameters dynamically, before the pipeline is executed (triggered from either the API or a webhook—a push event to your VCS).

Can I use a custom executor?

Custom executors can be used, but require certain dependencies to be installed for the continuation step to work (currently: curl, jq).

What is the continuation orb?

The continuation orb assists you in managing the pipeline continuation process. The continuation orb wraps an API call to continuePipeline. Refer to the continuation orb documentation for more information.

Is it possible to not use the continuation orb?

If you have special requirements not covered by the continuation orb, you can implement the same functionality in other ways. Refer to the orb source code to learn how the continuation functionality is implemented with the orb.