Configuring CircleCI

This document is a reference for the CircleCI 2.x configuration keys that are used in the config.yml file. The presence of a .circleci/config.yml file in your CircleCI-authorized repository branch indicates that you want to use the 2.x infrastructure.

You can see a complete config.yml in our full example.

Note: If you already have a CircleCI 1.0 configuration, the config.yml file allows you to test 2.x builds on a separate branch, leaving any existing configuration in the old circle.yml style unaffected and running on the CircleCI 1.0 infrastructure in branches that do not contain .circleci/config.yml.

Table of Contents

• version
• orbs (requires version: 2.1)
• commands (requires version: 2.1)
• parameters (requires version: 2.1)
• executors (requires version: 2.1)
• jobs
  • <job_name>
    • environment
    • parallelism
    • parameters
    • docker / machine / macos / windows (executor)
      • Available machine images
      • Available Linux GPU images
      • Available Windows GPU image
    • branches – DEPRECATED
    • resource_class
      • Docker Executor
        • Example Usage
      • Machine Executor (Linux)
        • Example Usage
      • macOS Executor
        • Example Usage
      • Windows Executor
        • Example Usage
      • GPU Executor (Linux)
        • Example Usage
      • GPU Executor (Windows)
        • Example Usage
    • steps
      • run
        • Default shell options
        • Background commands
        • Shorthand syntax
        • The when Attribute
        • Ending a Job from within a step
        • Example
      • The when Step (requires version: 2.1)
        • Example
      • checkout
      • setup_remote_docker
      • save_cache
        • Example
      • restore_cache
        • Example
      • deploy – DEPRECATED
        • Example
      • store_artifacts
        • Example
      • store_test_results
        • Example
      • persist_to_workspace
      • Example for root Key
        • Example for paths Key
      • attach_workspace
        • Example
      • add_ssh_keys
      • Using pipeline. Values
• workflows
  • version
  • <workflow_name>
    • triggers
      • schedule
        • cron
        • filters
    • jobs
      • <job_name>
        • requires
        • context
        • type
        • filters
        • matrix (requires version: 2.1)
        • pre-steps and post-steps (requires version: 2.1)
      • Using when in Workflows
• Logic Statements
  • Logic Statement Examples

version

The version field is intended to be used in order to issue warnings for deprecation or breaking changes.

orbs (requires version: 2.1)

The following example calls an Orb named hello-build that exists in the certified circleci namespace.

version: 2.1
orbs:
    hello: circleci/hello-build@0.0.5
workflows:
    "Hello Workflow":
        jobs:
          - hello/hello-build

In the above example, hello is considered the orbs reference; whereas circleci/hello-build@0.0.5 is the fully-qualified orb reference.

commands (requires version: 2.1)

A command definition defines a sequence of steps as a map to be executed in a job, enabling you to reuse a single command definition across multiple jobs.

Example:

commands:
  sayhello:
    description: "A very simple command for demonstration purposes"
    parameters:
      to:
        type: string
        default: "Hello World"
    steps:
      - run: echo << parameters.to >>

parameters (requires version: 2.1)

Pipeline parameters declared for use in the configuration. See Pipeline Variables for usage details.

executors (requires version: 2.1)

Executors define the environment in which the steps of a job will be run, allowing you to reuse a single executor definition across multiple jobs.

⁽¹⁾ One executor type should be specified per job. If more than one is set you will receive an error.

Example:

version: 2.1
executors:
  my-executor:
    docker:
      - image: circleci/ruby:2.5.1-node-browsers

jobs:
  my-job:
    executor: my-executor
    steps:
      - run: echo outside the executor

See the Using Parameters in Executors section of the Reusing Config document for examples of parameterized executors.

jobs

A run is comprised of one or more named jobs. Jobs are specified in the jobs map, see Sample 2.0 config.yml for two examples of a job map. The name of the job is the key in the map, and the value is a map describing the job.

If you are using Workflows, jobs must have unique names within the .circleci/config.yml file.

If you are not using workflows, the jobs map must contain a job named build. This build job is the default entry-point for a run that is triggered by a push to your VCS provider. It is possible to then specify additional jobs and run them using the CircleCI API.

Note: Jobs have a maximum runtime of 5 hours. If your jobs are timing out, consider running some of them concurrently using workflows.

<job_name>

Each job consists of the job’s name as a key and a map as a value. A name should be unique within a current jobs list. The value map has the following attributes:

⁽¹⁾ One executor type should be specified per job. If more than one is set you will receive an error.

environment

A map of environment variable names and values. These will override any environment variables you set in the CircleCI application.

parallelism

If parallelism is set to N > 1, then N independent executors will be set up and each will run the steps of that job in parallel. This can help optimize your test steps; you can split your test suite, using the CircleCI CLI, across parallel containers so the job will complete in a shorter time. Certain parallelism-aware steps can opt out of the parallelism and only run on a single executor (for example deploy step). Learn more about parallel jobs.

working_directory will be created automatically if it doesn’t exist.

Example:

jobs:
  build:
    docker:
      - image: buildpack-deps:trusty
    environment:
      FOO: bar
    parallelism: 3
    resource_class: large
    working_directory: ~/my-app
    steps:
      - run: go test -v $(go list ./... | circleci tests split)

parameters

The parameters can be used when calling that job in a workflow.

Reserved parameter-names:

• name
• requires
• context
• type
• filters
• matrix

See Parameter Syntax for definition details.

docker / machine / macos / windows (executor)

An “executor” is roughly “a place where steps occur”. CircleCI 2.0 can build the necessary environment by launching as many docker containers as needed at once, or it can use a full virtual machine. Learn more about different executors.

docker

Configured by docker key which takes a list of maps:

The first image listed in the file defines the primary container image where all steps will run.

entrypoint overrides the image’s ENTRYPOINT.

command overrides the image’s COMMAND; it will be used as arguments to the image ENTRYPOINT if it has one, or as the executable if the image has no ENTRYPOINT.

For a primary container (the first container in the list), if neither command nor entrypoint is specified in the config, then any ENTRYPOINT and COMMAND in the image are ignored. This is because the primary container is typically used only for running the steps and not for its ENTRYPOINT, and an ENTRYPOINT may consume significant resources or exit prematurely. (A custom image may disable this behavior and force the ENTRYPOINT to run.) The job steps run in the primary container only.

name defines the name for reaching the secondary service containers. By default, all services are exposed directly on localhost. The field is appropriate if you would rather have a different host name instead of localhost, for example, if you are starting multiple versions of the same service.

The environment settings apply to entrypoint/command run by the docker container, not the job steps.

You can specify image versions using tags or digest. You can use any public images from any public Docker registry (defaults to Docker Hub). Learn more about specifying images.

Example:

jobs:
  build:
    docker:
      - image: buildpack-deps:trusty # primary container
        environment:
          ENV: CI

      - image: mongo:2.6.8
        command: [--smallfiles]

      - image: postgres:9.4.1
        environment:
          POSTGRES_USER: root

      - image: redis@sha256:54057dd7e125ca41afe526a877e8bd35ec2cdd33b9217e022ed37bdcf7d09673

If you are using a private image, you can specify the username and password in the auth field. To protect the password, you can set it as a project setting which you reference here:

jobs:
  build:
    docker:
      - image: acme-private/private-image:321
        auth:
          username: mydockerhub-user  # can specify string literal values
          password: $DOCKERHUB_PASSWORD  # or project UI env-var reference

Using an image hosted on AWS ECR requires authentication using AWS credentials. By default, CircleCI uses the AWS credentials that you add to the Project > Settings > AWS Permissions page in the CircleCI application or by setting the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY project environment variables. It is also possible to set the credentials by using aws_auth field as in the following example:

jobs:
  build:
    docker:
      - image: account-id.dkr.ecr.us-east-1.amazonaws.com/org/repo:0.1
        aws_auth:
          aws_access_key_id: AKIAQWERVA  # can specify string literal values
          aws_secret_access_key: $ECR_AWS_SECRET_ACCESS_KEY  # or project UI envar reference

It is possible to reuse declared commands in a job when using version 2.1. The following example invokes the sayhello command.

jobs:
  myjob:
    docker:
      - image: "circleci/node:9.6.1"
    steps:
      - sayhello:
          to: "Lev"

machine

The machine executor is configured by using the machine key, which takes a map:

Example:

version: 2.1
jobs:
  build:
    machine:
      image: ubuntu-1604:202007-01
    steps:
      - checkout
      - run:
          name: "Testing"
          command: echo "Hi"

Available machine images

CircleCI supports multiple machine images that can be specified in the image field:

• ubuntu-1604:202007-01 - Ubuntu 16.04, Docker v19.03.12, Docker Compose v1.26.1
• ubuntu-1604:202004-01 - Ubuntu 16.04, Docker v19.03.8, Docker Compose v1.25.5
• ubuntu-1604:201903-01 - Ubuntu 16.04, Docker v18.09.3, Docker Compose v1.23.1
• circleci/classic:latest (old default) - an Ubuntu version 14.04 image that includes Docker version 17.09.0-ce and docker-compose version 1.14.0, along with common language tools found in CircleCI 1.0 build image. Changes to the latest image are announced at least a week in advance. Ubuntu 14.04 is now End-of-Life’d. We suggest using the Ubuntu 16.04 image.
• circleci/classic:edge - an Ubuntu version 14.04 image with Docker version 17.10.0-ce and docker-compose version 1.16.1, along with common language tools found in CircleCI 1.0 build image.
• circleci/classic:201703-01 – docker 17.03.0-ce, docker-compose 1.9.0
• circleci/classic:201707-01 – docker 17.06.0-ce, docker-compose 1.14.0
• circleci/classic:201708-01 – docker 17.06.1-ce, docker-compose 1.14.0
• circleci/classic:201709-01 – docker 17.07.0-ce, docker-compose 1.14.0
• circleci/classic:201710-01 – docker 17.09.0-ce, docker-compose 1.14.0
• circleci/classic:201710-02 – docker 17.10.0-ce, docker-compose 1.16.1
• circleci/classic:201711-01 – docker 17.11.0-ce, docker-compose 1.17.1
• circleci/classic:201808-01 – docker 18.06.0-ce, docker-compose 1.22.0

The machine executor supports Docker Layer Caching which is useful when you are building Docker images during your job or Workflow.

Available Linux GPU images

When using the Linux GPU executor, the available images are:

• ubuntu-1604-cuda-10.1:201909-23 - CUDA 10.1, docker 19.03.0-ce, nvidia-docker 2.2.2
• ubuntu-1604-cuda-9.2:201909-23 - CUDA 9.2, docker 19.03.0-ce, nvidia-docker 2.2.2

Available Windows GPU image

When using the Windows GPU executor, the available image is:

• windows-server-2019-nvidia:stable - Windows Server 2019, CUDA 10.1. This image is the default.

Example

version: 2.1
workflows:
  main:
    jobs:
      - build
jobs:
  build:
    machine:
      image: ubuntu-1604:201903-01
      docker_layer_caching: true    # default - false

macos

CircleCI supports running jobs on macOS, to allow you to build, test, and deploy apps for macOS, iOS, tvOS and watchOS. To run a job in a macOS virtual machine, you must add the macos key to the top-level configuration for the job and specify the version of Xcode you would like to use.

Example: Use a macOS virtual machine with Xcode version 11.3:

jobs:
  build:
    macos:
      xcode: "11.3.0"

windows

CircleCI supports running jobs on Windows. To run a job on a Windows machine, you must add the windows key to the top-level configuration for the job. Orbs also provide easy access to setting up a Windows job. To learn more about prerequisites to running Windows jobs and what Windows machines can offer, consult the Hello World on Windows document.

Example: Use a windows executor to run a simple job.

version: 2.1

orbs:
  win: circleci/windows@2.3.0

jobs:
  build:
    executor: win/default
    steps:
      - checkout
      - run: echo 'Hello, Windows'

branches – DEPRECATED

This key is deprecated. Use workflows filtering to control which jobs run for which branches.

Defines rules for allowing/blocking execution of some branches if Workflows are not configured and you are using 2.0 (not 2.1) config. If you are using Workflows, job-level branches will be ignored and must be configured in the Workflows section of your config.yml file. If you are using 2.1 config, you will need to add a workflow in order to use filtering. See the workflows section for details. The job-level branch key takes a map:

Both only and ignore lists can have full names and regular expressions. Regular expressions must match the entire string. For example:

jobs:
  build:
    branches:
      only:
        - master
        - /rc-.*/

In this case, only “master” branch and branches matching regex “rc-.*” will be executed.

jobs:
  build:
    branches:
      ignore:
        - develop
        - /feature-.*/

In this example, all the branches will be executed except “develop” and branches matching regex “feature-.*”.

If both ignore and only are present in config, only ignore will be taken into account.

A job that was not executed due to configured rules will show up in the list of jobs in UI, but will be marked as skipped.

To ensure the job runs for all branches, either don’t use the branches key, or use the only key along with the regular expression: /.*/ to catch all branches.

resource_class

The resource_class feature allows configuring CPU and RAM resources for each job. Different resource classes are available for different executors, as described in the tables below.

We implement soft concurrency limits for each resource class to ensure our system remains stable for all customers. If you are on a Performance or custom plan and experience queuing for certain resource classes, it’s possible you are hitting these limits. Contact CircleCI support to request a raise on these limits for your account.

Note: This feature is automatically enabled on free and Performance plans. Available resources classes are restricted for customers on the free plan to small/medium for linux, and medium for Windows. MacOS is not yet available on the free plan.

For self-hosted installations of CircleCI Server contact your system administrator for a list of available resource classes. See Server Administration documents for further information: Nomad Client System Requirements and Server Resource Classes.

Docker Executor

Example Usage

jobs:
  build:
    docker:
      - image: buildpack-deps:trusty
    resource_class: xlarge
    steps:
      ... // other config

Machine Executor (Linux)

Example Usage

jobs:
  build:
    machine:
      image: ubuntu-1604:201903-01 # recommended linux image - includes Ubuntu 16.04, docker 18.09.3, docker-compose 1.23.1
    resource_class: large
    steps:
      ... // other config

macOS Executor

Example Usage

jobs:
  build:
    macos:
      xcode: "11.3.0"
    resource_class: large
    steps:
      ... // other config

Windows Executor

Example Usage

version: 2.1

orbs:
  win: circleci/windows@2.3.0

jobs:
  build:
    executor:
      name: win/default
      size: "medium" # can be "medium", "large", "xlarge", "2xlarge"
    steps:
      - run: Write-Host 'Hello, Windows'

Note the way resource class is set is different for windows because the executor is defined within the windows orb.

See the Windows Getting Started document for more details and examples of using the Windows executor.

GPU Executor (Linux)

Example Usage

version: 2.1

jobs:
  build:
    machine:
      resource_class: gpu.nvidia.small
      image: ubuntu-1604-cuda-10.1:201909-23
    steps:
      - run: nvidia-smi
      - run: docker run --gpus all nvidia/cuda:9.0-base nvidia-smi

See the Available Linux GPU images section for the full list of available images.

GPU Executor (Windows)

Example Usage

version: 2.1
orbs:
  win: circleci/windows@2.3.0

jobs:
  build:
    executor: win/gpu-nvidia
    steps:
      - checkout
      - run: '&"C:\Program Files\NVIDIA Corporation\NVSMI\nvidia-smi.exe"'

⁽²⁾ This resource requires review by our support team. Open a support ticket if you would like to request access.

Note: Java, Erlang and any other languages that introspect the /proc directory for information about CPU count may require additional configuration to prevent them from slowing down when using the CircleCI 2.0 resource class feature. Programs with this issue may request 32 CPU cores and run slower than they would when requesting one core. Users of languages with this issue should pin their CPU count to their guaranteed CPU resources. If you want to confirm how much memory you have been allocated, you can check the cgroup memory hierarchy limit with grep hierarchical_memory_limit /sys/fs/cgroup/memory/memory.stat.

steps

The steps setting in a job should be a list of single key/value pairs, the key of which indicates the step type. The value may be either a configuration map or a string (depending on what that type of step requires). For example, using a map:

jobs:
  build:
    working_directory: ~/canary-python
    environment:
      FOO: bar
    steps:
      - run:
          name: Running tests
          command: make test

Here run is a step type. The name attribute is used by the UI for display purposes. The command attribute is specific for run step and defines command to execute.

Some steps may implement a shorthand semantic. For example, run may be also be called like this:

jobs:
  build:
    steps:
      - run: make test

In its short form, the run step allows us to directly specify which command to execute as a string value. In this case step itself provides default suitable values for other attributes (name here will have the same value as command, for example).

Another shorthand, which is possible for some steps, is to simply use the step name as a string instead of a key/value pair:

jobs:
  build:
    steps:
      - checkout

In this case, the checkout step will checkout project source code into the job’s working_directory.

In general all steps can be described as:

Each built-in step is described in detail below.

run

Used for invoking all command-line programs, taking either a map of configuration values, or, when called in its short-form, a string that will be used as both the command and name. Run commands are executed using non-login shells by default, so you must explicitly source any dotfiles as part of the command.

Each run declaration represents a new shell. It is possible to specify a multi-line command, each line of which will be run in the same shell:

- run:
    command: |
      echo Running test
      mkdir -p /tmp/test-results
      make test

You can also configure commands to run in the background if you don’t want to wait for the step to complete before moving on to subsequent run steps.

Default shell options

For jobs that run on Linux, the default value of the shell option is /bin/bash -eo pipefail if /bin/bash is present in the build container. Otherwise it is /bin/sh -eo pipefail. The default shell is not a login shell (--login or -l are not specified). Hence, the shell will not source your ~/.bash_profile, ~/.bash_login, ~/.profile files.

For jobs that run on macOS, the default shell is /bin/bash --login -eo pipefail. The shell is a non-interactive login shell. The shell will execute /etc/profile/ followed by ~/.bash_profile before every step.

For more information about which files are executed when bash is invocated, see the INVOCATION section of the bash manpage.

Descriptions of the -eo pipefail options are provided below.

-e

Exit immediately if a pipeline (which may consist of a single simple command), a subshell command enclosed in parentheses, or one of the commands executed as part of a command list enclosed by braces exits with a non-zero status.

So if in the previous example mkdir failed to create a directory and returned a non-zero status, then command execution would be terminated, and the whole step would be marked as failed. If you desire the opposite behaviour, you need to add set +e in your command or override the default shell in your configuration map of run. For example:

- run:
    command: |
      echo Running test
      set +e
      mkdir -p /tmp/test-results
      make test

- run:
    shell: /bin/sh
    command: |
      echo Running test
      mkdir -p /tmp/test-results
      make test

-o pipefail

If pipefail is enabled, the pipeline’s return status is the value of the last (rightmost) command to exit with a non-zero status, or zero if all commands exit successfully. The shell waits for all commands in the pipeline to terminate before returning a value.

For example:

- run: make test | tee test-output.log

If make test fails, the -o pipefail option will cause the whole step to fail. Without -o pipefail, the step will always run successfully because the result of the whole pipeline is determined by the last command (tee test-output.log), which will always return a zero status.

Note that even if make test fails the rest of pipeline will be executed.

If you want to avoid this behaviour, you can specify set +o pipefail in the command or override the whole shell (see example above).

In general, we recommend using the default options (-eo pipefail) because they show errors in intermediate commands and simplify debugging job failures. For convenience, the UI displays the used shell and all active options for each run step.

For more information, see the Using Shell Scripts document.

Background commands

The background attribute enables you to configure commands to run in the background. Job execution will immediately proceed to the next step rather than waiting for return of a command with the background attribute set to true. The following example shows the config for running the X virtual framebuffer in the background which is commonly required to run Selenium tests:

- run:
    name: Running X virtual framebuffer
    command: Xvfb :99 -screen 0 1280x1024x24
    background: true

- run: make test

Shorthand syntax

run has a very convenient shorthand syntax:

- run: make test

# shorthanded command can also have multiple lines
- run: |
    mkdir -p /tmp/test-results
    make test

In this case, command and name become the string value of run, and the rest of the config map for that run have their default values.

The when Attribute

By default, CircleCI will execute job steps one at a time, in the order that they are defined in config.yml, until a step fails (returns a non-zero exit code). After a command fails, no further job steps will be executed.

Adding the when attribute to a job step allows you to override this default behaviour, and selectively run or skip steps depending on the status of the job.

The default value of on_success means that the step will run only if all of the previous steps have been successful (returned exit code 0).

A value of always means that the step will run regardless of the exit status of previous steps. This is useful if you have a task that you want to run regardless of whether the previous steps are successful or not. For example, you might have a job step that needs to upload logs or code-coverage data somewhere.

A value of on_fail means that the step will run only if one of the preceding steps has failed (returns a non-zero exit code). It is common to use on_fail if you want to store some diagnostic data to help debug test failures, or to run custom notifications about the failure, such as sending emails or triggering alerts in chatrooms.

Note: Some steps, such as store_artifacts and store_test_results will always run, even if a step has failed previously.

- run:
    name: Upload CodeCov.io Data
    command: bash <(curl -s https://codecov.io/bash) -F unittests
    when: always # Uploads code coverage results, pass or fail

Ending a Job from within a step

A job can exit without failing by using using run: circleci-agent step halt. This can be useful in situations where jobs need to conditionally execute.

Here is an example where halt is used to avoid running a job on the develop branch:

run: |
    if [ "$CIRCLE_BRANCH" = "develop" ]; then
        circleci-agent step halt
    fi

Example

steps:
  - run:
      name: Testing application
      command: make test
      shell: /bin/bash
      working_directory: ~/my-app
      no_output_timeout: 30m
      environment:
        FOO: bar

  - run: echo 127.0.0.1 devhost | sudo tee -a /etc/hosts

  - run: |
      sudo -u root createuser -h localhost --superuser ubuntu &&
      sudo createdb -h localhost test_db

  - run:
      name: Upload Failed Tests
      command: curl --data fail_tests.log http://example.com/error_logs
      when: on_fail

The when Step (requires version: 2.1)

A conditional step consists of a step with the key when or unless. Under the when key are the subkeys condition and steps. The purpose of the when step is customizing commands and job configuration to run on custom conditions (determined at config-compile time) that are checked before a workflow runs. See the Conditional Steps section of the Reusing Config document for more details.

Example

version: 2.1

jobs: # conditional steps may also be defined in `commands:`
  job_with_optional_custom_checkout:
    parameters:
      custom_checkout:
        type: string
        default: ""
    machine: true
    steps:
      - when:
          condition: <<parameters.custom_checkout>>
          steps:
            - run: echo "my custom checkout"
      - unless:
          condition: <<parameters.custom_checkout>>
          steps:
            - checkout
workflows:
  build-test-deploy:
    jobs:
      - job_with_optional_custom_checkout:
          custom_checkout: "any non-empty string is truthy"
      - job_with_optional_custom_checkout

checkout

A special step used to check out source code to the configured path (defaults to the working_directory). The reason this is a special step is because it is more of a helper function designed to make checking out code easy for you. If you require doing git over HTTPS you should not use this step as it configures git to checkout over ssh.

If path already exists and is:

• a git repo - step will not clone whole repo, instead will pull origin
• NOT a git repo - step will fail.

In the case of checkout, the step type is just a string with no additional attributes:

- checkout

Note: CircleCI does not check out submodules. If your project requires submodules, add run steps with appropriate commands as shown in the following example:

- checkout
- run: git submodule sync
- run: git submodule update --init

This command will automatically add the required authenticity keys for interacting with GitHub and Bitbucket over SSH, which is detailed further in our integration guide – this guide will also be helpful if you wish to implement a custom checkout command.

Note: The checkout step will configure Git to skip automatic garbage collection. If you are caching your .git directory with restore_cache and would like to use garbage collection to reduce its size, you may wish to use a run step with command git gc before doing so.

setup_remote_docker

Creates a remote Docker environment configured to execute Docker commands. See Running Docker Commands for details.

Notes:

• A paid account on a Performance or Custom Plan is required to access Docker Layer Caching.
• setup_remote_docker is not compatible with the machine executor. See Docker Layer Caching in Machine Executor for information on how to enable DLC with the machine executor.
• The version key is not currently supported on CircleCI installed in your private cloud or datacenter. Contact your system administrator for information about the Docker version installed in your remote Docker environment.

save_cache

Generates and stores a cache of a file or directory of files such as dependencies or source code in our object storage. Later jobs can restore this cache. Learn more in the caching documentation.

The cache for a specific key is immutable and cannot be changed once written.

Note If the cache for the given key already exists it won’t be modified, and job execution will proceed to the next step.

When storing a new cache, the key value may contain special templated values for your convenience:

During step execution, the templates above will be replaced by runtime values and use the resultant string as the key.

Template examples:

• myapp-{{ checksum "package-lock.json" }} - cache will be regenerated every time something is changed in package-lock.json file, different branches of this project will generate the same cache key.
• myapp-{{ .Branch }}-{{ checksum "package-lock.json" }} - same as the previous one, but each branch will generate separate cache
• myapp-{{ epoch }} - every run of a job will generate a separate cache

While choosing suitable templates for your cache key, keep in mind that cache saving is not a free operation, because it will take some time to upload the cache to our storage. So it make sense to have a key that generates a new cache only if something actually changed and avoid generating a new one every run of a job.

Example

- save_cache:
    key: v1-myapp-{{ arch }}-{{ checksum "project.clj" }}
    paths:
      - /home/ubuntu/.m2

restore_cache

Restores a previously saved cache based on a key. Cache needs to have been saved first for this key using save_cache step. Learn more in the caching documentation.

⁽¹⁾ at least one attribute has to be present. If key and keys are both given, key will be checked first, and then keys.

A key is searched against existing keys as a prefix.

Note: When there are multiple matches, the most recent match will be used, even if there is a more precise match.

For example:

steps:
  - save_cache:
      key: v1-myapp-cache
      paths:
        - ~/d1

  - save_cache:
      key: v1-myapp-cache-new
      paths:
        - ~/d2

  - run: rm -f ~/d1 ~/d2

  - restore_cache:
      key: v1-myapp-cache

In this case cache v1-myapp-cache-new will be restored because it’s the most recent match with v1-myapp-cache prefix even if the first key (v1-myapp-cache) has exact match.

For more information on key formatting, see the key section of save_cache step.

When CircleCI encounters a list of keys, the cache will be restored from the first one matching an existing cache. Most probably you would want to have a more specific key to be first (for example, cache for exact version of package-lock.json file) and more generic keys after (for example, any cache for this project). If no key has a cache that exists, the step will be skipped with a warning.

A path is not required here because the cache will be restored to the location from which it was originally saved.

Example

- restore_cache:
    keys:
      - v1-myapp-{{ arch }}-{{ checksum "project.clj" }}
      # if cache for exact version of `project.clj` is not present then load any most recent one
      - v1-myapp-

# ... Steps building and testing your application ...

# cache will be saved only once for each version of `project.clj`
- save_cache:
    key: v1-myapp-{{ arch }}-{{ checksum "project.clj" }}
    paths:
      - /foo

deploy – DEPRECATED

This key is deprecated. For improved control over your deployments use workflows plus associated filtering and scheduling keys.

Special step for deploying artifacts.

deploy uses the same configuration map and semantics as run step. Jobs may have more than one deploy step.

In general deploy step behaves just like run with two exceptions:

• In a job with parallelism, the deploy step will only be executed by node #0 and only if all nodes succeed. Nodes other than #0 will skip this step.
• In a job that runs with SSH, the deploy step will not execute, and the following action will show instead:

  skipping deploy Running in SSH mode. Avoid deploying.

When using the deploy step, it is also helpful to understand how you can use workflows to orchestrate jobs and trigger jobs. For more information about using workflows, refer to the following pages:

• Workflows
• workflows

Example

- deploy:
    command: |
      if [ "${CIRCLE_BRANCH}" == "master" ]; then
        ansible-playbook site.yml
      fi

Note: The run step allows you to use a shortcut like run: my command; however, if you try to use a similar shortcut for the deploy step like deploy: my command, then you will receive the following error message in CircleCI:

In step 3 definition: This type of step does not support compressed syntax

store_artifacts

Step to store artifacts (for example logs, binaries, etc) to be available in the web app or through the API. See the Uploading Artifacts document for more information.

There can be multiple store_artifacts steps in a job. Using a unique prefix for each step prevents them from overwriting files.

Example

- run:
    name: Build the Jekyll site
    command: bundle exec jekyll build --source jekyll --destination jekyll/_site/docs/
- store_artifacts:
    path: jekyll/_site/docs/
    destination: circleci-docs

store_test_results

Special step used to upload and store test results for a build. Test results are visible on the CircleCI web application, under each build’s “Test Summary” section. Storing test results is useful for timing analysis of your test suites.

It is also possible to store test results as a build artifact; to do so, please refer to the store_artifacts step.

Note: Please write your tests to subdirectories of your store_test_results path, ideally named to match the names of your particular test suites, in order for CircleCI to correctly infer the names of your reports. If you do not write your reports to subdirectories, you will see reports in your “Test Summary” section such as Your build ran 71 tests in unknown, instead of, for example, Your build ran 71 tests in rspec.

Example

Directory structure:

test-results
├── jest
│   └── results.xml
├── mocha
│   └── results.xml
└── rspec
    └── results.xml

config.yml syntax:

- store_test_results:
    path: test-results

persist_to_workspace

Special step used to persist a temporary file to be used by another job in the workflow.

Note: Workspaces are stored for up to 15 days after being created. All jobs that try to use a Workspace older than 15 days, including partial reruns of a Workflow and SSH reruns of individual jobs, will fail.

The root key is a directory on the container which is taken to be the root directory of the workspace. The paths values are all relative to the root.

Example for root Key

For example, the following step syntax persists the specified paths from /tmp/dir into the workspace, relative to the directory /tmp/dir.

- persist_to_workspace:
    root: /tmp/dir
    paths:
      - foo/bar
      - baz

After this step completes, the following directories are added to the workspace:

/tmp/dir/foo/bar
/tmp/dir/baz

Example for paths Key

- persist_to_workspace:
    root: /tmp/workspace
    paths:
      - target/application.jar
      - build/*

The paths list uses Glob from Go, and the pattern matches filepath.Match.

pattern:
        { term }
term:
        '*' matches any sequence of non-Separator characters
        '?' matches any single non-Separator character
        '[' [ '^' ] { character-range }
        ']' character class (must be non-empty)
        c matches character c (c != '*', '?', '\\', '[')
        '\\' c matches character c
character-range:
        c matches character c (c != '\\', '-', ']')
        '\\' c matches character c
        lo '-' hi matches character c for lo <= c <= hi

The Go documentation states that the pattern may describe hierarchical names such as /usr/*/bin/ed (assuming the Separator is ‘/’). Note: Everything must be relative to the work space root directory.

attach_workspace

Special step used to attach the workflow’s workspace to the current container. The full contents of the workspace are downloaded and copied into the directory the workspace is being attached at.

Example

- attach_workspace:
    at: /tmp/workspace

Each workflow has a temporary workspace associated with it. The workspace can be used to pass along unique data built during a job to other jobs in the same workflow. Jobs can add files into the workspace using the persist_to_workspace step and download the workspace content into their file system using the attach_workspace step. The workspace is additive only, jobs may add files to the workspace but cannot delete files from the workspace. Each job can only see content added to the workspace by the jobs that are upstream of it. When attaching a workspace the “layer” from each upstream job is applied in the order the upstream jobs appear in the workflow graph. When two jobs run concurrently the order in which their layers are applied is undefined. If multiple concurrent jobs persist the same filename then attaching the workspace will error.

If a workflow is re-run it inherits the same workspace as the original workflow. When re-running failed jobs only the re-run jobs will see the same workspace content as the jobs in the original workflow.

Note the following distinctions between Artifacts, Workspaces, and Caches:

Refer to the Persisting Data in Workflows: When to Use Caching, Artifacts, and Workspaces for additional conceptual information about using workspaces, caching, and artifacts.

add_ssh_keys

Special step that adds SSH keys from a project’s settings to a container. Also configures SSH to use these keys.

steps:
  - add_ssh_keys:
      fingerprints:
        - "b7:35:a6:4e:9b:0d:6d:d4:78:1e:9a:97:2a:66:6b:be"

Note: Even though CircleCI uses ssh-agent to sign all added SSH keys, you must use the add_ssh_keys key to actually add keys to a container.

Using pipeline. Values

Pipeline values are available to all pipeline configurations and can be used without previous declaration. The pipeline values available are as follows:

For example:

version: 2.1
jobs:
  build:
    docker:
      - image: circleci/node:latest
    environment:
      IMAGETAG: latest
    working_directory: ~/main
    steps:
      - run: echo "This is pipeline ID << pipeline.id >>"

workflows

Used for orchestrating all jobs. Each workflow consists of the workflow name as a key and a map as a value. A name should be unique within the current config.yml. The top-level keys for the Workflows configuration are version and jobs.

version

The Workflows version field is used to issue warnings for deprecation or breaking changes during Beta.

<workflow_name>

A unique name for your workflow.

triggers

Specifies which triggers will cause this workflow to be executed. Default behavior is to trigger the workflow when pushing to a branch.

schedule

A workflow may have a schedule indicating it runs at a certain time, for example a nightly build that runs every day at 12am UTC:

workflows:
   version: 2
   nightly:
     triggers:
       - schedule:
           cron: "0 0 * * *"
           filters:
             branches:
               only:
                 - master
                 - beta
     jobs:
       - test

cron

The cron key is defined using POSIX crontab syntax.

filters

Filters can have the key branches.

branches

The branches key controls whether the current branch should have a schedule trigger created for it, where current branch is the branch containing the config.yml file with the trigger stanza. That is, a push on the master branch will only schedule a workflow for the master branch.

Branches can have the keys only and ignore which either map to a single string naming a branch. You may also use regular expressions to match against branches by enclosing them with /’s, or map to a list of such strings. Regular expressions must match the entire string.

• Any branches that match only will run the job.
• Any branches that match ignore will not run the job.
• If neither only nor ignore are specified then all branches will run the job.
• If both only and ignore are specified the only is considered before ignore.

jobs

A job can have the keys requires, context, type, and filters.

<job_name>

A job name that exists in your config.yml.

requires

Jobs are run in parallel by default, so you must explicitly require any dependencies by their job name.

context

Jobs may be configured to use global environment variables set for an organization, see the Contexts document for adding a context in the application settings.

type

A job may have a type of approval indicating it must be manually approved before downstream jobs may proceed. Jobs run in the dependency order until the workflow processes a job with the type: approval key followed by a job on which it depends, for example:

      - hold:
          type: approval
          requires:
            - test1
            - test2
      - deploy:
          requires:
            - hold

Note: The hold job name must not exist in the main configuration.

filters

Filters can have the key branches or tags. Note Workflows will ignore job-level branching. If you use job-level branching and later add workflows, you must remove the branching at the job level and instead declare it in the workflows section of your config.yml, as follows:

The following is an example of how the CircleCI documentation uses a regex to filter running a workflow for building PDF documentation:

# ...
workflows:
  build-deploy:
    jobs:
      - js_build
      - build_server_pdfs: # << the job to conditionally run based on the filter-by-branch-name.
          filters:
            branches:
              only: /server\/.*/

The above snippet causes the job build_server_pdfs to only be run when the branch being built contains the word “server/” in it.

You can read more about using regex in your config in the Workflows document.

branches

Branches can have the keys only and ignore which either map to a single string naming a branch. You may also use regular expressions to match against branches by enclosing them with slashes, or map to a list of such strings. Regular expressions must match the entire string.

• Any branches that match only will run the job.
• Any branches that match ignore will not run the job.
• If neither only nor ignore are specified then all branches will run the job.
• If both only and ignore are specified the only is considered before ignore.

tags

CircleCI does not run workflows for tags unless you explicitly specify tag filters. Additionally, if a job requires any other jobs (directly or indirectly), you must specify tag filters for those jobs.

Tags can have the keys only and ignore keys. You may also use regular expressions to match against tags by enclosing them with slashes, or map to a list of such strings. Regular expressions must match the entire string. Both lightweight and annotated tags are supported.

• Any tags that match only will run the job.
• Any tags that match ignore will not run the job.
• If neither only nor ignore are specified then the job is skipped for all tags.
• If both only and ignore are specified the only is considered before ignore.

For more information, see the Executing Workflows For a Git Tag section of the Workflows document.

matrix (requires version: 2.1)

The matrix stanza allows you to run a parameterized job multiple times with different arguments.

The following is a basic example of using matrix jobs.

workflows:
  workflow:
    jobs:
      - build:
          matrix:
            parameters:
              version: ["0.1", "0.2", "0.3"]
              platform: ["macos", "windows", "linux"]

This expands to 9 different build jobs, and could be equivalently written as:

workflows:
  workflow:
    jobs:
      - build:
          name: build-macos-0.1
          version: 0.1
          platform: macos
      - build:
          name: build-macos-0.2
          version: 0.2
          platform: macos
      - build:
          name: build-macos-0.3
          version: 0.3
          platform: macos
      - build:
          name: build-windows-0.1
          version: 0.1
          platform: windows
      - ...

Excluding sets of parameters from a matrix

Sometimes you may wish to run a job with every combination of arguments except some value or values. You can use an exclude stanza to achieve this:

workflows:
  workflow:
    jobs:
      - build:
          matrix:
            parameters:
              a: [1, 2, 3]
              b: [4, 5, 6]
            exclude:
              - a: 3
                b: 5

The matrix above would expand into 8 jobs: every combination of the parameters a and b, excluding {a: 3, b: 5}

Dependencies and matrix jobs

To require an entire matrix (every job within the matrix), use its alias. The alias defaults to the name of the job being invoked.

workflows:
  workflow:
    jobs:
      - deploy:
          matrix:
            parameters:
              version: ["0.1", "0.2"]
      - another-job:
          requires:
            - deploy

This means that another-job will require both deploy jobs in the matrix to finish before it runs.

Additionally, matrix jobs expose their parameter values via << matrix.* >> which can be used to generate more complex workflows. For example, here is a deploy matrix where each job waits for its respective build job in another matrix.

workflows:
  workflow:
    jobs:
      - build:
          name: build-v<< matrix.version >>
          matrix:
            parameters:
              version: ["0.1", "0.2"]
      - deploy:
          name: deploy-v<< matrix.version >>
          matrix:
            parameters:
              version: ["0.1", "0.2"]
          requires:
            - build-v<< matrix.version >>

This workflow will expand to:

workflows:
  workflow:
    jobs:
      - build:
          name: build-v0.1
          version: 0.1
      - build:
          name: build-v0.2
          version: 0.2
      - deploy:
          name: deploy-v0.1
          version: 0.1
          requires:
            - build-v0.1
      - deploy:
          name: deploy-v0.2
          version: 0.2
          requires:
            - build-v0.2

pre-steps and post-steps (requires version: 2.1)

Every job invocation in a workflow may optionally accept two special arguments: pre-steps and post-steps.

Steps under pre-steps are executed before any of the other steps in the job. The steps under post-steps are executed after all of the other steps.

Pre and post steps allow you to execute steps in a given job without modifying the job. This is useful, for example, to run custom setup steps before job execution.

version: 2.1

jobs:
  bar:
    machine: true
    steps:
      - checkout
      - run:
          command: echo "building"
      - run:
          command: echo "testing"

workflows:
  build:
    jobs:
      - bar:
          pre-steps: # steps to run before steps defined in the job bar
            - run:
                command: echo "install custom dependency"
          post-steps: # steps to run after steps defined in the job bar
            - run:
                command: echo "upload artifact to s3"

Using when in Workflows

With CircleCI v2.1 configuration, you may use a when clause (the inverse clause unless is also supported) under a workflow declaration with a logic statement to determine whether or not to run that workflow.

The example configuration below uses a pipeline parameter, run_integration_tests to drive the integration_tests workflow.

version: 2.1

parameters:
  run_integration_tests:
    type: boolean
    default: false

workflows:
  integration_tests:
    when: << pipeline.parameters.run_integration_tests >>
    jobs:
      - mytestjob

jobs:
...

This example prevents the workflow integration_tests from running unless the tests are invoked explicitly when the pipeline is triggered with the following in the POST body:

{
    "parameters": {
        "run_integration_tests": true
    }
}

Refer to the Orchestrating Workflows document for more examples and conceptual information.

Logic Statements

Certain dynamic configuration features accept logic statements as arguments. Logic statements are evaluated to boolean values at configuration compilation time, that is - before the workflow is run. The group of logic statements includes:

Truthiness rules are as follows:

false, null, 0, the empty string, and NaN are falsy. Any other value is truthy.

Logic statements always evaluate to a boolean value at the top level, and coerce as necessary. They can be nested in an arbitrary fashion, according to their argument specifications, and to a maximum depth of 100 levels.

Logic Statement Examples

workflows:
  my-workflow:
      when:
        condition:
          or:
            - equal: [ master, << pipeline.git.branch >> ]
            - equal: [ staging, << pipeline.git.branch >> ]

workflows:
  my-workflow:
    when:
      and:
        - not:
            equal: [ master, << pipeline.git.branch >> ]
        - or:
            - equal: [ canary, << pipeline.git.tag >> ]
            - << pipeline.parameters.deploy-canary >>

Example Full Configuration

version: 2
jobs:
  build:
    docker:
      - image: ubuntu:14.04

      - image: mongo:2.6.8
        command: [mongod, --smallfiles]

      - image: postgres:9.4.1
        # some containers require setting environment variables
        environment:
          POSTGRES_USER: root

      - image: redis@sha256:54057dd7e125ca41afe526a877e8bd35ec2cdd33b9217e022ed37bdcf7d09673

      - image: rabbitmq:3.5.4

    environment:
      TEST_REPORTS: /tmp/test-reports

    working_directory: ~/my-project

    steps:
      - checkout

      - run:
          command: echo 127.0.0.1 devhost | sudo tee -a /etc/hosts

      # Create Postgres users and database
      # Note the YAML heredoc '|' for nicer formatting
      - run: |
          sudo -u root createuser -h localhost --superuser ubuntu &&
          sudo createdb -h localhost test_db

      - restore_cache:
          keys:
            - v1-my-project-{{ checksum "project.clj" }}
            - v1-my-project-

      - run:
          environment:
            SSH_TARGET: "localhost"
            TEST_ENV: "linux"
          command: |
            set -xu
            mkdir -p ${TEST_REPORTS}
            run-tests.sh
            cp out/tests/*.xml ${TEST_REPORTS}

      - run: |
          set -xu
          mkdir -p /tmp/artifacts
          create_jars.sh ${CIRCLE_BUILD_NUM}
          cp *.jar /tmp/artifacts

      - save_cache:
          key: v1-my-project-{{ checksum "project.clj" }}
          paths:
            - ~/.m2

      # Save artifacts
      - store_artifacts:
          path: /tmp/artifacts
          destination: build

      # Upload test results
      - store_test_results:
          path: /tmp/test-reports

  deploy-stage:
    docker:
      - image: ubuntu:14.04
    working_directory: /tmp/my-project
    steps:
      - run:
          name: Deploy if tests pass and branch is Staging
          command: ansible-playbook site.yml -i staging

  deploy-prod:
    docker:
      - image: ubuntu:14.04
    working_directory: /tmp/my-project
    steps:
      - run:
          name: Deploy if tests pass and branch is Master
          command: ansible-playbook site.yml -i production

workflows:
  version: 2
  build-deploy:
    jobs:
      - build:
          filters:
            branches:
              ignore:
                - develop
                - /feature-.*/
      - deploy-stage:
          requires:
            - build
          filters:
            branches:
              only: staging
      - deploy-prod:
          requires:
            - build
          filters:
            branches:
              only: master

See Also

Config Introduction