Writing Jobs with Steps

This document describes how to write jobs and steps to build, test, and deploy your project. The presence of a .circleci/config.yml file in your CircleCI-authorized repository branch indicates that you want to use the 2.0 infrastructure.

If you already have a CircleCI 1.0 configuration, the config.yml file allows you to test 2.0 builds on a separate branch, leaving any existing configuration in the old circle.yml style unaffected and running on the CircleCI 1.0 infrastructure in branches that do not contain .circleci/config.yml.

You can see a complete config.yml in our full example.

Table of Contents

• version
• jobs
  • build
    • docker | machine (executor)
    • branches
    • resource_class
    • steps
      • run
      • Default shell options
      • Background commands
      • Shorthand syntax
      • The when Attribute
      • Example
      • checkout
      • save_cache
        • Example
      • restore_cache
        • Example
      • deploy
        • Example
      • store_artifacts
        • Example
      • store_test_results
        • Example
      • persist_to_workspace
        • Example
      • attach_workspace
        • Example
      • add_ssh_keys
• workflows
  • version
  • jobs
    • build
      • requires
      • filters
        • branches
      • Example

version

The version field is intended to be used in order to issue warnings for deprecation or breaking changes.

jobs

Each job is an item in the jobs list.

build

The build job is the default job and is required if you are not using Workflows. If you are using Workflows, a job named build is optional. See the Workflows documentation for detailed information.

Each job consists of the job’s name as a key and a map as a value. A name should be unique within a current jobs list. The value map has the following attributes:

⁽¹⁾ exactly one of them should be specified. It is an error to set more than one.

If parallelism is set to N > 1, then N independent executors will be set up and each will run the steps of that job in parallel. Certain parallelism-aware steps can opt out of the parallelism and only run on a single executor (for example deploy step). Learn more about parallel jobs.

working_directory will be created automatically if it doesn’t exist.

Example:

jobs:
  build:
    docker:
      - buildpack-deps:trusty
    environment:
      - FOO: "bar"
    parallelism: 3
    resource_class: large
    working_directory: ~/my-app
    branches:
      only:
        - master
        - /rc-.*/
    steps:
      - run: make test
      - run: make

docker | machine (executor)

An “executor” is roughly “a place where steps occur”. CircleCI 2.0 can build the necessary environment by launching as many docker containers as needed at once, or it can use a full virtual machine. Learn more about different executors.

docker

Configured by docker key which takes a list of maps:

The first image listed in the file defines the primary container image where all steps will run.

entrypoint overrides default entrypoint from Dockerfile.

command will be used as arguments to image entrypoint (if specified in Dockerfile) or as executable (if no entrypoint is provided here or in the Dockerfile).

For primary container (listed first in the list) if no command is specified then command and image entrypoint will be ignored, to avoid errors caused by the entrypoint executable consuming significant resources or exiting prematurely. At this time all steps run in the primary container only.

The environment settings apply to all commands run in this executor, not just the initial command. The environment here has higher precedence over setting it in the job map above.

You can specify image versions using tags or digest. You can use any public images from any public Docker registry (defaults to Docker Hub). Learn more about specifying images.

Example:

jobs:
  build:
    docker:
      - image: buildpack-deps:trusty # primary container
        environment:
          ENV: CI

      - image: mongo:2.6.8
        command: [--smallfiles]

      - image: postgres:9.4.1
        environment:
          POSTGRES_USER: root

      - image: redis@sha256:54057dd7e125ca41afe526a877e8bd35ec2cdd33b9217e022ed37bdcf7d09673

machine

The usage of the machine executor is configured by using the machine key, which takes a map:

As a shorthand, you can set the machine key to true.

Example:

jobs:
  build:
    machine:
      enabled: true

# or just

jobs:
  build:
    machine: true

CircleCI supports multiple machine images that can be specified in image field:

• circleci/classic:latest (default) - an Ubuntu Trusty-based image with Docker 17.03.0-ce along with common language tools found in CircleCI 1.0 build image. The latest channel provides the latest tested images, changes to the channel are announced at least a week in advance.
• circleci/classic:edge - an Ubuntu Trusty-based image with Docker 17.06.0-ce along with common language tools found in CircleCI 1.0 build image. The edge channel provides release candidates that will eventually be promoted to classic:latest.

So you can set the following to use Docker 17.06.0-ce:

jobs:
  build:
    machine:
      image: circleci/classic:edge

branches

Defines rules for whitelisting/blacklisting execution of some branches if Workflows are not configured. If you are using Workflows, job-level branches will be ignored and must be configured in the Workflows section of your ‘config.yml’ file. See the workflows section for details. The job-level branch key takes a map:

Both only and ignore lists can have full names and regular expressions. For example:

branches:
  only:
    - master
    - /rc-.*/

In this case only “master” branch and branches matching regex “rc-.*” will be executed.

branches:
  ignore:
    - develop
    - /feature-.*/

In this example all the branches will be executed except “develop” and branches matching regex “feature-.*”.

If both ignore and only are present in config, only ignore will be taken into account.

A job that was not executed due to configured rules will show up in the list of jobs in UI, but will be marked as skipped.

resource_class

It is possible to configure CPU and RAM resources for each job as described in the following table. If resource_class is not specified or an invalid class is specified, the default resource_class: medium will be used. The resource_class key is currently only available for use with the docker executor. Paid accounts may request this feature from their Customer Success Manager, non-paid users may request to get started by sending email to support@circleci.com.

steps

The steps setting in a job should be a list of single key/value pairs, the key of which indicates the step type. The value may be either a configuration map or a string (depending on what that type of step requires). For example, using a map:

jobs:
  build:
    working_directory: ~/canary-python
    environment:
      - FOO: "bar"
    steps:
      - run:
          name: Running tests
          command: make test

Here run is a step type. The name attribute is used by the UI for display purposes. The command attribute is specific for run step and defines command to execute.

Some steps may implement a shorthand semantic. For example, run may be also be called like this:

jobs:
  build:
    steps:
      - run: make test

In its short form, the run step allows us to directly specify which command to execute as a string value. In this case step itself provides default suitable values for other attributes (name here will have the same value as command, for example).

Another shorthand, which is possible for some steps, is to simply use the step name as a string instead of a key/value pair:

jobs:
  build:
    steps:
      - checkout

In this case, the checkout step will checkout project source code into the job’s working_directory.

In general all steps can be describe as:

Each built-in step is described in detail below.

run

Used for invoking all command-line programs, taking either a map of configuration values, or, when called in its short-form, a string that will be used as both the command and name. Run commands are executed using non-login shells by default, so you must explicitly source any dotfiles as part of the command.

Each run declaration represents a new shell. It’s possible to specify a multi-line command, each line of which will be run in the same shell:

- run:
    command: |
      echo Running test
      mkdir -p /tmp/test-results
      make test

Default shell options

Our default shell has a few options enabled by default:

-e

Exit immediately if a pipeline (which may consist of a single simple command), a subshell command enclosed in parentheses, or one of the commands executed as part of a command list enclosed by braces exits with a non-zero status.

So if in the previous example mkdir failed to create a directory and returned a non-zero status, then command execution would be terminated, and the whole step would be marked as failed. If you desire the opposite behaviour, you need to add set +e in your command or override the default shell in your configuration map of run. For example:

- run:
    command: |
      echo Running test
      set +e
      mkdir -p /tmp/test-results
      make test

- run:
    shell: /bin/sh
    command: |
      echo Running test
      mkdir -p /tmp/test-results
      make test

-o

If pipefail is enabled, the pipeline’s return status is the value of the last (rightmost) command to exit with a non-zero status, or zero if all commands exit successfully. The shell waits for all commands in the pipeline to terminate before returning a value.

For example:

- run: make test | tee test-output.log

If make test fails, the -o pipefail option will cause the whole step to fail. Without -o pipefail, the step will always run successfully because the result of the whole pipeline is determined by the last command (tee test-output.log), which will always return a zero status.

Note that even if make test fails the rest of pipeline will be executed.

If you want to avoid this behaviour, you can specify set +o pipefail in the command or override the whole shell (see example above).

In general, we recommend using the default options (-eo pipefail) because they show errors in intermediate commands and simplify debugging job failures. For convenience, the UI displays the used shell and all active options for each run step.

Background commands

The background attribute allows for executing commands in the background. In this case, job execution will immediately proceed to the next step rather than waiting for the command to return. While debugging background commands is more difficult, running commands in the background might be necessary in some cases. For instance, to run Selenium tests you may need to have X virtual framebuffer running:

- run:
    name: Running X virtual framebuffer
    command: Xvfb :99 -screen 0 1280x1024x24
    background: true

- run: make test

Shorthand syntax

run has a very convenient shorthand syntax:

- run: make test

# shorthanded command can also have multiple lines
- run: |
    mkdir -p /tmp/test-results
    make test

In this case, command and name become the string value of run, and the rest of the config map for that run have their default values.

The when Attribute

By default, CircleCI will execute job steps one at a time, in the order that they are defined in config.yml, until a step fails (returns a non-zero exit code). After a command fails, no further job steps will be executed.

Adding the when attribute to a job step allows you to override this default behaviour, and selectively run or skip steps depending on the status of the job.

The default value of on_success means that the step will run only if all of the previous steps have been successful (returned exit code 0).

A value of always means that the step will run regardless of the exit status of previous steps. This is useful if you have a task that you want to run regardless of whether the previous steps are successful or not. For example, you might have a job step that needs to upload logs or code-coverage data somewhere.

A value of on_fail means that the step will run only if one of the preceding steps has failed (returns a non-zero exit code). It is common to use on_fail if you want to store some diagnostic data to help debug test failures, or to run custom notifications about the failure, such as sending emails or triggering alerts in chatrooms.

Example

- run:
    name: Testing application
    command: make test
    shell: /bin/bash
    working_directory: ~/my-app
    no_output_timeout: 30m
    environment:
      FOO: "bar"

- run: echo 127.0.0.1 devhost | sudo tee -a /etc/hosts

- run: |
    sudo -u root createuser -h localhost --superuser ubuntu &&
    sudo createdb -h localhost test_db

- run:
    name: Upload Failed Tests
    command: curl --data fail_tests.log http://example.com/error_logs
    when: on_fail

checkout

Special step used to check out source code to the configured path (defaults to the working_directory).

If path already exists and is:

• a git repo - step will not clone whole repo, instead will pull origin
• NOT a git repo - step will fail.

In the case of checkout, the step type is just a string with no additional attributes:

- checkout

save_cache

Generates and stores a cache of a file or directory of files such as dependencies or source code in our object storage. Later jobs can restore this cache. Learn more in the caching documentation.

The cache for a specific key is immutable and cannot be changed once written. NOTE: If the cache for the given key already exists it won’t be modified, and job execution will proceed to the next step.

When storing a new cache, the key value may contain special templated values for your convenience:

During step execution, the templates above will be replaced by runtime values and use the resultant string as the key.

Template examples:

• myapp-{{ checksum "package.json" }} - cache will be regenerated every time something is changed in package.json file, different branches of this project will generate the same cache key.
• myapp-{{ .Branch }}-{{ checksum "package.json" }} - same as the previous one, but each branch will generate separate cache
• myapp-{{ epoch }} - every run of a job will generate a separate cache

While choosing suitable templates for your cache key, keep in mind that cache saving is not a free operation, because it will take some time to upload the cache to our storage. So it make sense to have a key that generates a new cache only if something actually changed and avoid generating a new one every run of a job.

Example

- save_cache:
    key: v1-myapp-{{ checksum "project.clj" }}
    paths:
      - /home/ubuntu/.m2

restore_cache

Restores a previously saved cache based on a key. Cache needs to have been saved first for this key using save_cache step. Learn more in the caching documentation.

⁽¹⁾ at least one attribute has to be present. If key and keys are both given, key will be checked first, and then keys.

A key is searched against existing keys as a prefix.

NOTE: When there are multiple matches, the most recent match will be used, even if there is a more precise match.

For example:

steps:
  - save_cache:
      key: v1-myapp-cache
      paths:
        - ~/d1

  - save_cache:
      key: v1-myapp-cache-new
      paths:
        - ~/d2

  - run: rm -f ~/d1 ~/d2

  - restore_cache:
      key: v1-myapp-cache

In this case cache v1-myapp-cache-new will be restored because it’s the most recent match with v1-myapp-cache prefix even if the first key (v1-myapp-cache) has exact match.

For more information on key formatting, see the key section of save_cache step.

When CircleCI encounters a list of keys, the cache will be restored from the first one matching an existing cache. Most probably you would want to have a more specific key to be first (for example, cache for exact version of package.json file) and more generic keys after (for example, any cache for this project). If no key has a cache that exists, the step will be skipped with a warning.

A path is not required here because the cache will be restored to the location from which it was originally saved.

Example

- restore_cache:
    keys:
      - v1-myapp-{{ checksum "project.clj" }}
      # if cache for exact version of `project.clj` is not present then load any most recent one
      - v1-myapp-

# ... Steps building and testing your application ...

# cache will be saved only once for each version of `project.clj`
- save_cache:
    key: v1-myapp-{{ checksum "project.clj" }}
    paths:
      - /foo

deploy

Special step for deploying artifacts.

deploy uses the same configuration map and semantics as run step. Jobs may have more than one deploy step.

In general deploy step behaves just like run with one exception - in a job with parallelism, the deploy step will only be executed by node #0 and only if all nodes succeed. Nodes other than #0 will skip this step.

Example

- deploy:
    command: |
      if [ "${CIRCLE_BRANCH}" == "master" ]; then
        ansible-playbook site.yml
      fi

store_artifacts

Step to store artifacts (for example logs, binaries, etc) to be available in the web app or through the API. See the Uploading Artifacts document for more information.

There can be multiple store_artifacts steps in a job. Using a unique prefix for each step prevents them from overwriting files.

Example

- store_artifacts:
    path: /code/test-results
    destination: prefix

store_test_results

Special step used to upload test results so they can be used for timing analysis. Note At this time the results are not shown as artifacts in the web UI. To see test result as artifacts please also upload them using store_artifacts. This key is not supported with Workflows.

The directory layout should match the classic CircleCI test metadata directory layout.

Example

- store_test_results:
    path: /tmp/test-results

persist_to_workspace

Special step used to persist a temporary file to be used by another job in the workflow.

Example

- persist_to_workspace:
    root: /tmp/workspace
    paths: 
      - target/application.jar
      - build/*

attach_workspace

Special step used to attach the workflow’s workspace to the current container. The full contents of the workspace are downloaded and copied into the directory the workspace is being attached at.

Example

- attach_workspace:
    at: /tmp/workspace

Each workflow has a temporary workspace associated with it. The workspace can be used to pass along unique data built during a job to other jobs in the same workflow. Jobs can add files into the workspace using the persist_to_workspace step and download the workspace content into their fileystem using the attach_workspace step. The workspace is additive only, jobs may add files to the workspace but cannot delete files from the workspace. Each job can only see content added to the workspace by the jobs that are upstream of it.

When attaching a workspace the “layer” from each upstream job is applied in the order the upstream jobs appear in the workflow graph. When two jobs run concurrently the order in which their layers are applied is undefined. If multiple concurrent jobs persist the same filename then attaching the workspace will error.

If a workflow is re-run it inherits the same workspace as the original workflow. When re-running failed jobs only the re-run jobs will see the same workspace content as the jobs in the original workflow.

Note the following distinctions between Artifacts, Workspaces, and Caches:

add_ssh_keys

Special step that adds SSH keys configured in the project’s UI to the container, and configure ssh to use them.

- add_ssh_keys:
    fingerprints:
      - "b7:35:a6:4e:9b:0d:6d:d4:78:1e:9a:97:2a:66:6b:be"

Note that CircleCI 2.0 jobs are auto configured with ssh-agent with all keys auto-loaded, and is sufficient for most cases. add_ssh_keys may be needed to have greater control over which SSH keys to authenticate (e.g. avoid “Too many authentication failures” problem when having too many SSH keys).

workflows

Used for orchestrating all jobs. Each workflow consists of the workflow name as a key and a map as a value. A name should be unique within the current config.yml. The top-level keys for the Workflows configuration are version and jobs.

version

The Workflows version field is used to issue warnings for deprecation or breaking changes during Beta.

jobs

A job can have the keys requires and filters.

build

A unique name for your job.

requires

Jobs are run in parallel by default, so you must explicitly require any dependencies by their job name.

filters

Filters can have the key branches. Note Workflows will ignore job-level branching. If you use job-level branching and later add workflows, you must remove the branching at the job level and instead declare it in the workflows section of your config.yml, as follows:

branches

Branches can have the keys only and ignore which either map to a single string naming a branch (or a regexp to match against branches, which is required to be enclosed with /s) or map to a list of such strings.

• Any branches that match only will run the job.
• Any branches that match ignore will not run the job.
• If neither only nor ignore are specified then all branches will run the job.
• If both only and ignore are specified the only overrides ignore.

Example

workflows:
  version: 2

  build_test_deploy:
    jobs:
      - flow
      - downstream:
          requires:
            - flow
          filters:
            branches:
              only: master

Refer to the Orchestrating Workflows document for more examples and conceptual information.

Full Example

version: 2
jobs:
  build:
    docker:
      - image: ubuntu:14.04

      - image: mongo:2.6.8
        command: [mongod, --smallfiles]

      - image: postgres:9.4.1
        # some containers require setting environment variables
        environment:
          POSTGRES_USER: root

      - image: redis@sha256:54057dd7e125ca41afe526a877e8bd35ec2cdd33b9217e022ed37bdcf7d09673

      - image: rabbitmq:3.5.4

    environment:
      TEST_REPORTS: /tmp/test-reports

    working_directory: ~/my-project

    branches:
      ignore:
        - develop
        - /feature-.*/

    steps:
      - checkout

      - run: command: echo 127.0.0.1 devhost | sudo tee -a /etc/hosts

      # Create Postgres users and database
      # Note the YAML heredoc '|' for nicer formatting
      - run: |
          sudo -u root createuser -h localhost --superuser ubuntu &&
          sudo createdb -h localhost test_db

      - restore_cache:
          keys:
            - v1-my-project-{{ checksum "project.clj" }}
            - v1-my-project-

      - run:
          environment:
            SSH_TARGET: "localhost"
            TEST_ENV: "linux"
          command: |
            set -xu
            mkdir -p ${TEST_REPORTS}
            run-tests.sh
            cp out/tests/*.xml ${TEST_REPORTS}

      - run: |
          set -xu
          mkdir -p /tmp/artifacts
          create_jars.sh ${CIRCLE_BUILD_NUM}
          cp *.jar /tmp/artifacts

      - save_cache:
          key: v1-my-project-{{ checksum "project.clj" }}
          paths:
            - ~/.m2

      # Deploy staging
      - deploy:
          command: |
            if [ "${CIRCLE_BRANCH}" == "staging" ];
              then ansible-playbook site.yml -i staging;
            fi

      # Deploy production
      - deploy:
          command: |
            if [ "${CIRCLE_BRANCH}" == "master" ];
              then ansible-playbook site.yml -i production;
            fi

      # Save artifacts
      - store_artifacts:
          path: /tmp/artifacts
          destination: build

      # Upload test results
      - store_test_results:
          path: /tmp/test-reports