Using dynamic configuration

1 month ago9 min read

Cloud

Server v4.x

Server v3.x

This page covers some dynamic configuration examples. The following examples are provided below:

Continue a pipeline with a dynamically generated configuration file: In this example, a script is used to generate a YAML configuration file. The continuation orb is used to continue the pipeline using the generated configuration.
Execute specific workflows or steps based on which files are modified: In this example, two configuration files are used, a setup configuration and a continuation configuration. The path filtering orb is used to conditionally set pipeline parameters, and conditional workflow logic is used to generate and continue the pipeline based on the location of modified files.
Generate a configuration file based on modified files: In this example the continuation configuration is split into separate project-specific config files and a shared config file. Path filtering is used to generate a config file for continuation based on the location of modified files.
Pack, generate, and validate a configuration file for pipeline continuation: In this example the continuation configuration is split into separate project-specific config files, and a shared config directory. Path filtering is used to pack, generate and validate a config file for continuation based on the location of modified files.

Prerequisites

The content on this page assumes you have already read the Dynamic Configuration page.
If your project was created before December 1st 2023 you will need to enable dynamic configuration in your project settings.

Continue a pipeline with a dynamically generated configuration file

The following is a basic example using CircleCI’s dynamic configuration feature.

In this example, we assume that a generate-config script already exists. The script outputs a new configuration YAML based on some type of work it performs, for example, inspecting git history or pipeline values that get passed to it, or anything else you might do from inside a job.

version: 2.1

# define this file as the setup phase of your dynamic configuration
setup: true

# invoke the continuation orb to make the continuation/continue command available
orbs:
  continuation: circleci/continuation@1

jobs:
  setup:
    executor: continuation/default
    steps:
      - checkout # checkout code
      - run: # run command to run script to generate YAML config
          name: Generate config
          command: |
            ./generate-config > generated_config.yml
      - continuation/continue:
          configuration_path: generated_config.yml # use newly generated config to continue the pipeline

workflows:
  my-setup-workflow:
    jobs:
      - setup

Line 4: Add the line setup: true to the top-level of our config, to designate it for use of CircleCI’s dynamic configuration feature.
Line 8: Invoke the continuation orb so we can use it.
Lines 11-20: Define a job called setup that uses the default executor provided by the continuation orb as an . This job:
- Calls the checkout step to checkout code from the configured repository.
- Calls the run step to execute the existing generate-config script, so we can pass its output to the continue job of the continuation orb.
- Continues running the pipeline based on what configuration is provided to the required configuration_path.
Line 25: We call the setup job defined above as a part of our workflow.

Only one workflow that includes a continuation is allowed per pipeline. To adhere to this you can either only configure a single continuation workflow, or you can use conditional logic to ensure only one will run in a pipeline. This setup-workflow has access to a one-time-use token to create more workflows. The setup process does not cascade, therefore subsequent workflows in the pipeline cannot launch their own continuations.

For a more in-depth explanation of what the continuation orb does, review the orb’s source code in the CircleCI developer hub.

Execute specific `workflows` or `steps` based on which files are modified

Your project may benefit from the ability to conditionally run some configuration based upon changes made to a specific set of files. Dynamically deciding the work to do in a pipeline based on the location of changes is beneficial when your code/microservices are stored in a monorepo, or a single repository. Without this ability, using static configuration, each time a change is made to any file, all jobs and workflows would run even if they are unrelated to the actual change being made.

Use the path filtering orb to help simplify the process of using dynamic configuration based on the paths to modified files.

Consider a monorepo structure, as follows:

.
├── .circleci
│   ├── config.yml
│   └── continue_config.yml
├── service1
│   ├── Service1.java
├── service2
│   ├── Service2.java
├── tests
│   ├── IntegrationTests.java

An example implementation of CircleCI’s dynamic configuration for this structure is provided in the following config.yml and continue_config.yml.

Setup configuration: `config.yml`

The config.yml file defines the initial setup phase of the dynamic configuration. In this example the path filtering orb is used to map file paths with pipeline parameter values. This means, if a change is made to a file matching a path regex, the corresponding parameter is set to the given value. These pipeline parameters are then passed to the continuation configuration and are used in the configured workflow conditional logic to determine which jobs should run.

The path filtering orb uses the continuation orb under the hood to continue the pipeline with the modified parameter values. The configuration used to continue the pipeline is described in the next section.

version: 2.1

# define this file as the setup phase of your dynamic configuration
setup: true

# use the path-filtering orb to continue a pipeline based on
# the path of an updated set of files
orbs:
  path-filtering: circleci/path-filtering@1

workflows:
  always-run:
    jobs:
      # the path-filtering/filter job determines which pipeline
      # parameters to update.
      - path-filtering/filter:
          name: check-updated-files
          # 3-column, whitespace-delimited mapping. One mapping per
          # line:
          # <regex path-to-test> <parameter-to-set> <value-of-pipeline-parameter>
          mapping: |
            service1/.* run-build-service-1-job true
            service2/.* run-build-service-2-job true
          base-revision: main
          # this is the path of the configuration we should trigger once
          # path filtering and pipeline parameter value updates are
          # complete.
          config-path: .circleci/continue_config.yml # this is the default so not actually required but left in to illustrate options

Continuation configuration: `continue_config.yml`

In this example, continue_config.yml is the continuation configuration, which means it is run once the initial config.yml finishes executing the path-filtering/filter job. The continuation configuration takes the updated pipeline parameter values, which were modified based on the paths to any changes in a commit, and uses them to conditionally run workflows using the when key. For more information on using when in workflows, see the Configuration reference.

version: 2.1

orbs:
  maven: circleci/maven@1.2.0

# the default pipeline parameters, which will be updated according to
# the results of the path-filtering orb
parameters:
  run-build-service-1-job:
    type: boolean
    default: false
  run-build-service-2-job:
    type: boolean
    default: false

# here we specify our workflows, most of which are conditionally
# executed based upon pipeline parameter values. Each workflow calls a
# specific job. In this example all jobs are preconfigured in the Maven orb
workflows:
  # when pipeline parameter run-build-service-1-job is true, the
  # build-service-1 job is triggered.
  service-1:
    when: << pipeline.parameters.run-build-service-1-job >>
    jobs:
      - maven/test:
          name: build-service-1
          command: 'install -DskipTests'
          app_src_directory: 'service1'
  # when pipeline parameter run-build-service-2-job is true, the
  # build-service-2 job is triggered.
  service-2:
    when: << pipeline.parameters.run-build-service-2-job >>
    jobs:
      - maven/test:
          name: build-service-2
          command: 'install -DskipTests'
          app_src_directory: 'service2'
  # when pipeline parameter, run-build-service-1-job OR
  # run-build-service-2-job is true, run-integration-tests job is
  # triggered. see:
  # https://circleci.com/docs/configuration-reference/#logic-statements
  # for more information.
  run-integration-tests:
    when:
      or: [<< pipeline.parameters.run-build-service-1-job >>, << pipeline.parameters.run-build-service-2-job >>]
    jobs:
      - maven/test:
          name: run-integration-tests
          command: '-X verify'
          app_src_directory: 'tests'

Line 4: Invoke the Maven orb so we can use it’s preconfigured jobs
Lines 8-14: Define our two boolean pipeline parameters, the same parameters we have defined in the setup phase: run-build-service-1-job and run-build-service-2-job
Define three separate workflows to be conditionally executed based on the pipeline parameter values:
- Lines 22-28: The service-1 workflow triggers the maven/test job on the service-1 directory, when the pipeline parameter value mapped to run-build-service-1-job is set to true. This will only happen if a change was made in the serivce-1 directory in the commit, as determined based on the path filtering in the setup phase (config.yml).
- Lines 31-37: The service-2 workflow triggers the maven/test job on the service-2 directory, when the pipeline parameter value mapped to run-build-service-2-job is set to true. This will only happen if a change was made in the serivce-2 directory in the commit, as determined based on the path filtering in the setup phase (config.yml).
- Lines 43-50: The run-integration-tests workflow will run if the run-build-service-1-job or run-build-service-2-job pipeline parameters have been updated to true based on the results of the path filtering. This runs the maven/test job on the tests directory to run integration tests against the built services.

Generate a configuration file based on modified files

This project is available on GitHub and you can see it building on CircleCI.

For this example, consider a project that includes:

Separate directories for code (src/) and docs (docs/).
A setup configuration, config.yml.
Separate configuration files for building the code and the docs, code-config.yml and docs-config.yml.
A shared-config.yml file that defines shared jobs to be used when building the code and the docs.
A no-updates.yml config file to be used in the event that a pipeline is triggered with no changes.

Each configuration file is explained in the following sections.

.
├── .circleci
│   ├── code-config.yml
│   ├── config.yml
│   ├── docs-config.yml
│   ├── no-updates.yml
│   └── shared-config.yml
├── README.md
├── docs
│   └── my-docs.txt
└── src
    └── my-code.txt

Setup configuration

In this example, the setup configuration includes a single job referenced from the path filtering orb, which is used to map pipeline parameter values and configuration files with paths to specific locations in the repository.

Under the hood the following steps are taken when the pipeline is triggered:

A script runs to check for modified files in a commit against a base branch (in this example, the default, generate-config-file-main).
A continuation configuration is generated using the relevant config files from the project, and the pipeline parameter values are set.
The pipeline is continued using the continuation configuration.

version: 2.1

# define this file as the setup phase of your dynamic configuration
setup: true

# invoke the path-filtering orb to make the filter job available.
orbs:
  path-filtering: circleci/path-filtering@1.0.0

workflows:
  setup-workflow:
    jobs:
      - path-filtering/filter:
          base-revision: generate-config-file-main
          config-path: .circleci/no-updates.yml
          mapping: | # The mapping will be used to generate the dynamic configuration for all conditions that match.
            .* always-continue true .circleci/shared-config.yml
            src/.* build-code true .circleci/code-config.yml
            docs/.* build-docs true .circleci/docs-config.yml

Shared configuration

In the event of changes to any file in the repository (.*) the shared-config.yml configuration is included in the continuation configuration. The only time shared-config.yml will not be used is on a commit that contains no changes.

shared-config.yml is designed to be used either on its own or along with other partial configurations. You can include anything that should always run in the shared configuration.. code-config.yml or docs-config.yml include workflows that orchestrate the shared jobs defined in shared-config.yml and they can be combined together to create the required pipeline for a set of changes.

version: 2.1

# define the parameters from the setup config.
parameters:
  always-continue:
    type: boolean
    default: false
  build-code:
    type: boolean
    default: false
  build-docs:
    type: boolean
    default: false

# define the shared jobs that will be available for all continued workflows.
jobs:
  lint:
    docker:
      - image: cimg/base:stable
    steps:
      - run: echo "Running linting"

  test:
    docker:
      - image: cimg/base:stable
    steps:
      - run: echo "Running tests"

  any-change:
    docker:
      - image: cimg/base:stable
    steps:
      - run: echo "This is a shared job that will reun for any change in the project."

workflows:
  run-on-any-change:
    jobs:
      - any-change

Directory-specific configuration

The config files for building the separate directories, src/ and /docs, contain build jobs for the content and workflows to orchestrate the build and shared jobs described above. The lint job is used for both, and the test job is only used when the code is built, not the docs.

`code-config.yml`

version: 2.1

jobs:
  build-code:
    docker:
      - image: cimg/base:stable
    steps:
      - run: echo "Building code"

workflows:
  code-workflow:
    jobs:
      - lint # use the shared lint job.
      - test # use the shared test job.
      - build-code

`docs-config.yml`

version: 2.1

jobs:
  build-docs:
    docker:
      - image: cimg/base:stable
    steps:
      - run: echo "Building docs"

workflows:
  docs-workflow:
    jobs:
      - lint # use the shared lint job.
      - build-docs

Configuration for no change

We have used the path filtering orb to map configuration files with file paths so we know which configuration will be used when a change is made in a specific location. If a pipeline is triggered with no changes in any of our defined paths, CircleCI needs to know what to do next.

One way to handle this scenario is to provide an alternative configuration using the config-path parameter (see line 27 here). config-path is ignored if a mapping is in place, but will be used if no mapping matches the path to modified files. In this example, we use a no-updates.yml configuration file. Another option would be to configure a step using the finish command from the continuation orb.

version: 2.1

# define the parameters from the setup config.
parameters:
  always-continue:
    type: boolean
    default: false
  build-code:
    type: boolean
    default: false
  build-docs:
    type: boolean
    default: false

jobs:
  no-updates:
    docker:
      - image: cimg/base:stable
    steps:
      - run: echo "No updates have been made"

workflows:
  no-update-workflow:
    jobs:
      - no-updates

Pack, generate, and validate a configuration file for pipeline continuation

This project is available on GitHub and you can see it building on CircleCI.

For this example, consider a project that includes:

Separate directories for code (src/) and docs (docs/).
A setup configuration, config.yml.
Separate configuration files for building the code and the docs, code-config.yml and docs-config.yml.
A shared/ configuration directory containing our shared jobs in separate YAML files. This is used to generate shared-config.yml using the CircleCI CLI command circleci config pack
A no-updates.yml config file to be used in the event that there are no changes.

.
├── .circleci
│   ├── code-config.yml
│   ├── config.yml
│   ├── docs-config.yml
│   ├── no-updates.yml
│   ├── shared
│   │   └── jobs
│   │       ├── any-change.yml
│   │       ├── lint.yml
│   │       └── test.yml
│   │   └── workflows
│   │       ├── run-on-any-change.yml
|   |       └── @shared.yml
├── README.md
├── docs
│   └── my-docs.txt
└── src
    └── my-code.txt

@shared.yml exists to provide the required version key for the config in the event that a pipeline is triggered on a change outside the src/ or docs/ directories.

Setup configuration

The setup configuration had a single job, setup, which has the following steps:

Checkout code
Install the CircleCI CLI
Generate a shared configuration file
Map file paths with pipeline parameter values and configuration files, using the path filtering orb
Generate a configuration file to fit the changes included in the commit that triggered the pipeline
Validate the configuration file using the CLI
Continue the pipeline with the new configuration file using the continuation orb

version: 2.1

# define this file as the setup phase of your dynamic configuration
setup: true

# define the parameters that will be used to generate the dynamic configuration.
parameters:
  always-continue:
    type: boolean
    default: false
  build-code:
    type: boolean
    default: false
  build-docs:
    type: boolean
    default: false

# invoke the orbs to filter, pack and continue configs.
orbs:
  path-filtering: circleci/path-filtering@1.0.0
  circleci-cli: circleci/circleci-cli@0.1.9
  continuation: circleci/continuation@1.0.0

jobs:
  setup:
    executor: path-filtering/default
    steps:
      - checkout

      # Install the CircleCI CLI
      - circleci-cli/install

      # Generate the shared configuration from a directory with the pack command.
      - run:
          name: Generate shared configuration
          command: circleci config pack .circleci/shared >> .circleci/shared-config.yml

      # The mapping will be used to generate the dynamic configuration for all conditions that match.
      - path-filtering/set-parameters:
          base-revision: pack-validate-continue-main
          config-path: .circleci/no-updates.yml
          mapping: |
            .* always-continue true .circleci/shared-config.yml
            src/.* build-code true .circleci/code-config.yml
            docs/.* build-docs true .circleci/docs-config.yml

      # Generate the dynamic configuration based on the parameters set in the previous step.
      - path-filtering/generate-config

      # Optionally validate the generated configuration.
      - run:
          name: Validate config
          command: circleci config validate /tmp/generated-config.yml

      # Continue the pipeline with the generated configuration.
      - continuation/continue:
          configuration_path: /tmp/generated-config.yml

workflows:
  setup-workflow:
    jobs:
      - setup

Shared configuration

In this example, shared configuration elements are stored in individual YAML files using a directory structure compatible with the circleci config pack command.

Jobs are stored in a jobs folder, workflows stored in a workflows folder, and so on. These files can be used in the setup stage of our dynamic configuration to generate a shared configuration by "packing" the relevant components together.

When packing a config using circleci config pack, the component names are not included in the configuration files, they are taken from the file names.

`shared/workflows/run-on-any-change.yml`

This workflow runs the job any-change on all pipelines triggered by a change in the repository, even if that change is outside of the /src or /docs directories.

jobs:
  - any-change

`shared/jobs/any-change.yml`

docker:
  - image: cimg/base:stable
steps:
  - run: echo "This is a shared job that will run for any change in the project."

`shared/jobs/lint.yml`

docker:
  - image: cimg/base:stable
steps:
  - run: echo "Running linting"

`shared/jobs/test.yml`

docker:
  - image: cimg/base:stable
steps:
  - run: echo "Running tests"

Directory-specific configuration

`code-config.yml`

version: 2.1

jobs:
  build-code:
    docker:
      - image: cimg/base:stable
    steps:
      - run: echo "Building code"

workflows:
  code-workflow:
    jobs:
      - lint # use the shared lint job.
      - test # use the shared test job.
      - build-code

`docs-config.yml`

version: 2.1

jobs:
  build-docs:
    docker:
      - image: cimg/base:stable
    steps:
      - run: echo "Building docs"

workflows:
  docs-workflow:
    jobs:
      - lint # use the shared lint job.
      - build-docs

Configuration for no change

version: 2.1

# define the parameters from the setup config.
parameters:
  always-continue:
    type: boolean
    default: false
  build-code:
    type: boolean
    default: false
  build-docs:
    type: boolean
    default: false

jobs:
  no-updates:
    docker:
      - image: cimg/base:stable
    steps:
      - run: echo "No updates have been made"

workflows:
  no-update-workflow:
    jobs:
      - no-updates

Another option would be to configure a step using the finish command from the continuation orb.

Suggest an edit to this page

Make a contribution

Learn how to contribute

Still need help?

Ask the CircleCI community

Join the research community

Visit our Support site

Using dynamic configuration

On This Page

Prerequisites

Continue a pipeline with a dynamically generated configuration file

Execute specific workflows or steps based on which files are modified

Setup configuration: config.yml

Continuation configuration: continue_config.yml

Generate a configuration file based on modified files

Setup configuration

Shared configuration

Directory-specific configuration

code-config.yml

docs-config.yml

Configuration for no change

Pack, generate, and validate a configuration file for pipeline continuation

Setup configuration

Shared configuration

shared/workflows/run-on-any-change.yml

shared/jobs/any-change.yml

shared/jobs/lint.yml

shared/jobs/test.yml

Directory-specific configuration

code-config.yml

docs-config.yml

Configuration for no change

Suggest an edit to this page

Still need help?

Execute specific `workflows` or `steps` based on which files are modified

Setup configuration: `config.yml`

Continuation configuration: `continue_config.yml`

`code-config.yml`

`docs-config.yml`

`shared/workflows/run-on-any-change.yml`

`shared/jobs/any-change.yml`

`shared/jobs/lint.yml`

`shared/jobs/test.yml`

`code-config.yml`

`docs-config.yml`