Search Results for ""

Reusing Config

This document describes how to version your .circleci/config.yml file and get started with reusable commands and executors.

Getting Started with Config Reuse

  1. Add your project on the Add Projects page if it is a new project. For an existing Project, go to the Project Settings and enable Build Processing with the radio button.

  2. (Optional) Install the CircleCI-Public CLI by following the Using the CircleCI CLI documentation. The circleci config process command is helpful for checking reusable config.

  3. Change the version key to 2.1 in your .circleci/config.yml file and commit the changes to test your build. Ensure that your project build succeeds with the new processing before adding any new 2.1 keys to your config.

  4. Run builds with your new configuration by pushing to your GitHub or Bitbucket repo that has been added in a project in CircleCI. The Jobs page displays runs using the new processing service.

After your build is running successfully with build processing enabled and version 2.1 in the .circleci/config.yml file, it is possible to add new keys to reuse config and run the same job more than once with different parameters (re-use jobs).

Authoring Reusable Commands

A reusable command may have the following immediate children keys as a map:

  • Description: (optional) A string that describes the purpose of the command, used for generating documentation.
  • Parameters: (optional) A map of parameter keys, each of which adheres to the parameter spec.
  • Steps: (required) A sequence of steps run inside the calling job of the command.

Commands are declared under the commands key of a config.yml file. The following example defines a command called sayhello:

commands:
  sayhello:
    description: "A very simple command for demonstration purposes"
    parameters:
      to:
        type: string
        default: "Hello World"
    steps:
      - run: echo << parameters.to >>

Note: The commands stanza is available in configuration version 2.1 and later.

Invoking Reusable Commands

Reusable commands are invoked with specific parameters as steps inside a job. When using a command, the steps of that command are inserted in the location where the command is invoked. Commands may only be used as part of the sequence under steps in a job.

The following example invokes the command sayhello and passes it a parameter to:

jobs:
  myjob:
    docker:
      - image: "circleci/node:9.6.1"
    steps:
      - sayhello:
          to: "Lev"

Authoring Reusable Executors

Executors define the environment in which the steps of a job will be run. When declaring a job in CircleCI configuration, you define the type of execution environment (docker, machine, macos. etc.) to run in, as well as any other parameters of that environment including: environment variables to populate, which shell to use, what size resource_class to use, etc.

Executor declarations in config outside of jobs can be used by all jobs in the scope of that declaration, allowing you to reuse a single executor definition across multiple jobs.

An executor definition includes one or more of the following keys:

  • docker or machine or macos
  • environment
  • working_directory
  • shell
  • resource_class

In the following example my-executor is passed as the single value of the key executor.

version: 2.1
executors:
  my-executor:
    docker:
      - image: circleci/ruby:2.5.1-node-browsers

jobs:
  my-job:
    executor: my-executor
    steps:
      - run: echo outside the executor

Note: Reusable executor declarations are available in configuration version 2.1 and later.

Invoking Reusable Executors

The following example passes my-executor as the value of a name key under executor – this method is primarily employed when passing parameters to executor invocations (see below):

jobs:
  my-job:
    executor:
      name: my-executor
    steps:
      - run: echo outside the executor

Example of Using an Executor Declared in config.yml in Multiple Jobs

The following example declares and invokes an executor in two jobs that need to run in the same Docker image and working directory with a common set of environment variables. Each job has distinct steps, but runs in the same environment.

version: 2.1
executors:
  lein_exec: # declares a reusable executor
    docker:
      - image: clojure:lein-2.8.1
    working_directory: ~/project
    environment:
      MYSPECIALVAR: "my-special-value"
      MYOTHERVAR: "my-other-value"
    
jobs:
  build:
    executor: lein_exec
    steps:
      - checkout
      - run: echo "hello world"
      
  test:
    executor: lein_exec
    environment:
      TESTS: unit
    steps:
      - checkout
      - run: echo "how are ya?"

Overriding Keys When Invoking an Executor

When invoking an executor in a job any keys in the job itself will override those of the executor invoked. For example, if your job declares a docker stanza, it will be used, in its entirety, instead of the one in your executor.

Note: The environment variable maps are additive. If an executor has one of the same environment variables as the job, the value in the job will be used. For example, if you had the following configuration:

executors:
  python:
    docker:
      - image: python:3.7.0
      - image: rabbitmq:3.6-management-alpine
    environment:
      ENV: ci
      TESTS: all
    shell: /bin/bash    
    working_directory: ~/project

jobs:
  build:
    docker:
      - image: python:2.7.15
      - image: postgres:9.6
    executor: python
    environment:
      TESTS: unit
    working_directory: ~/tests

The above config would resolve to the following:

jobs:
 build:
   steps: []
   docker:
     - image: python:2.7.15    # From job
     - image: postgres:9.6     # From job
   environment:                # Merged:
     ENV: ci                     # From executor
     TESTS: unit                 # From job
   shell: /bin/bash            # From executor
   working_directory: ~/tests  # From job

Using the parameters Declaration

Parameters are declared by name under a job, command, or executor. The immediate children of the parameters key are a set of keys in a map.

The following example defines a command called s3sync:

commands: # a reusable command with parameters
  s3sync:
    parameters:
      from:
        type: string
      to:
        type: string
      overwrite:
        default: false
        type: boolean
    steps:
      - run: # a parameterized run step
          name: Deploy to S3
          command: "aws s3 sync << parameters.from >> << parameters.to >><<# parameters.overwrite >> --delete<</ parameters.overwrite >>"
executors: # a reusable executor
  aws:
    docker:
      - image: cibuilds/aws:1.15
jobs: # a job that invokes the `aws` executor and the `s3sync` command
  deploy2s3:
    executor: aws
    steps:
      - s3sync:
          from: .
          to: "s3://mybucket_uri"
          overwrite: true

Note: The parameters declaration is available in configuration version 2.1 and later.

Parameter Syntax

A parameter can have the following keys as immediate children:

Key Name Description Default value
description Optional. Used to generate documentation for your orb. N/A
type Required. Currently “string”, “boolean”, “enum” (takes extra keys), and “steps” are supported N/A
default The default value for the parameter. If not present, the parameter is implied to be required. N/A

Parameter Types

This section describes the types of parameters and their usage.

Parameter Scope

Parameters are in-scope only within the job or command that defined them. If you want a job or command to pass its parameters to a command it invokes, they must be passed explicitly. Command, job, executor, and parameter names can only contain lowercase letters a-z, digits, and _ and -, and must start with a letter.

version: 2.1

jobs:
  sayhello:
    parameters:
      saywhat:
        description: "To whom shall we say hello?"
        default: "World"
        type: string
    machine: true
    steps:
      - say:
          # Since the command "say" doesn't define a default
          # value for the "saywhat" parameter, it must be
          # passed in manually
          saywhat: << parameters.saywhat >>

commands:
  say:
    parameters:
      saywhat:
        type: string
    steps:
      - run: echo "<< parameters.saywhat >>"

workflows:
  build:
    jobs:
      - sayhello:
          saywhat: Everyone

String

Basic string parameters are described below:

commands:
  copy-markdown:
    parameters:
      destination:
        description: destination directory
        type: string
        default: docs
    steps:
      - cp *.md << destination >>

Strings should be quoted if they would otherwise represent another type (such as boolean or number) or if they contain characters that have special meaning in YAML, particularly for the colon character. In all other instances, quotes are optional. Empty strings are treated as a falsy value in evaluation of when clauses, and all other strings are treated as truthy. Using an unquoted string value that YAML interprets as a boolean will result in a type error.

Boolean

Boolean parameters are useful for conditionals:

commands:
  list-files:
    parameters:
      all:
        description: include all files
        type: boolean
        default: false
    steps:
      - ls <<# all >> -a <</ all >>

Boolean parameter evaluation is based on the values specified in YAML 1.1:

  • True: y yes true on
  • False: n no false off

Capitalized and uppercase versions of the above values are also valid.

Steps

Steps are used when you have a job or command that needs to mix predefined and user-defined steps. When passed in to a command or job invocation, the steps passed as parameters are always defined as a sequence, even if only one step is provided.

commands:
  run-tests:
    parameters:
      after-deps:
        description: "Steps that will be executed after dependencies are installed, but before tests are run"
        type: steps
        default: []
    steps:
    - run: make deps
    - steps: << parameters.after-deps >>
    - run: make test

The following example demonstrates that steps passed as parameters are given as the value of a steps declaration under the job’s steps.

jobs:
  build:
    machine: true
    steps:
    - run-tests:
        after-deps:
          - run: echo "The dependencies are installed"
          - run: echo "And now I'm going to run the tests"

The above will resolve to the following:

steps:
  - run: make deps
  - run: echo "The dependencies are installed"
  - run: echo "And now I'm going to run the tests"
  - run: make test

Enum Parameter

The enum parameter may be a list of any values. Use the enum parameter type when you want to enforce that the value must be one from a specific set of string values. The following example uses the enum parameter to declare the target operating system for a binary.

commands:
  list-files:
    parameters:
      os: 
        default: "linux"
        description: The target Operating System for the heroku binary. Must be one of "linux", "darwin", "win32".
        type: enum
        enum: ["linux", "darwin", "win32"]

The following enum type declaration is invalid because the default is not declared in the enum list.

commands:
  list-files:
    parameters:      
      os:
        type: enum
        default: "windows" #invalid declaration of default that does not appear in the comma-separated enum list
        enum: ["darwin", "linux"]

Using Parameters in Executors

To use parameters in executors, define the parameters under the given executor. When you invoke the executor, pass the keys of the parameters as a map of keys under the executor: declaration, each of which has the value of the parameter to pass in.

Parameters in executors can be of the type string, enum, or boolean. Default values can be provided with the optional default key.

Example Build Configuration Using a Parameterized Executor

version: 2.1

executors:
  python:
    parameters:
      tag:
        type: string
        default: latest
      myspecialvar:
        type: string
    docker:
      - image: circleci/python:<< parameters.tag >>
    environment:
      MYPRECIOUS: << parameters.myspecialvar >>

jobs:
  build:
    executor:
      name: python
      tag: "2.7"
      myspecialvar: "myspecialvalue"  

The above would resolve to the following:

version: 2.1
jobs:
  build:
    steps: []
    docker:
      - image: circleci/python:2.7
    environment:
      MYPRECIOUS: "myspecialvalue"

Authoring Parameterized Jobs

It is possible to invoke the same job more than once in the workflows stanza of config.yml, passing any necessary parameters as subkeys to the job. See the parameters section above for details of syntax usage.

Example of defining and invoking a parameterized job in a config.yml:

version: 2.1

jobs:
  sayhello: # defines a parameterized job
    description: A job that does very little other than demonstrate what a parameterized job looks like
    parameters:
      saywhat:
        description: "To whom shall we say hello?"
        default: "World"
        type: string
    machine: true
    steps:
      - run: echo "Hello << parameters.saywhat >>"

workflows:
  build:
    jobs:
      - sayhello: # invokes the parameterized job
          saywhat: Everyone

Note: Invoking jobs multiple times in a single workflow and parameters in jobs are available in configuration version 2.1 and later.

Invoking the Same Job Multiple Times

A single configuration may invoke a job multiple times. At configuration processing time during build ingestion, CircleCI will auto-generate names if none are provided or you may name the duplicate jobs explicitly with the name key.

Note: You must explicitly name repeat jobs when a repeat job should be upstream of another job in a workflow. For example, if a job is used under the requires key of a job invocation in a workflow you will need to explicitly name it.

version: 2.1
workflows:
  build:
    jobs:
      - loadsay
      # This doesn't need an explicit name as it has no downstream dependencies
      - sayhello:
          saywhat: Everyone
          requires:
            - loadsay
      # This needs an explicit name for saygoodbye to require it as a job dependency
      - sayhello:
          name: SayHelloChad
          saywhat: Chad
      # Uses explicitly defined "sayhello"
      - saygoodbye:
          requires:
            - SayHelloChad

Using Pre and Post Steps

Every job invocation may optionally accept two special arguments: pre-steps and post-steps. Steps under pre-steps are executed before any of the other steps in the job. The steps under post-steps are executed after all of the other steps.

Pre and post steps allow you to execute steps in a given job without modifying the job. This is useful, for example, to run custom setup steps before job execution.

Defining Pre and Post Steps

The following example defines pre-steps and post-steps in the bar job of the build workflow:

# config.yml
version: 2.1
jobs:
  bar:
    machine: true
    steps:
      - checkout
      - run:
          command: echo "building"
      - run:
          command: echo "testing"
workflows:
  build:
    jobs:
      - bar:
          pre-steps:
            - run:
                command: echo "install custom dependency"
          post-steps:
            - run:
                command: echo "upload artifact to s3"

Note: The keys pre-steps and post-steps in jobs are available in configuration version 2.1 and later.

Defining Conditional Steps

Conditional steps allow the definition of steps that only run if a condition is met.

These conditions are checked before a workflow is actually run. That means, for example, that a you cannot use a condition to check an environment variable.

Conditional steps may be located anywhere a regular step could and may only use parameter values as inputs.

A conditional step consists of a step with the key when or unless. Under this conditional key are the subkeys steps and condition. If condition is met (using when/unless logic), the subkey steps are run.

A condition is a single value that evaluates to true or false at the time the config is processed, so you cannot use environment variables as conditions, as those are not injected until your steps are running in the shell of your execution environment. You may use parameters as your conditions. The empty string will resolve as falsey in when conditions.

Example

# inside config.yml
version: 2.1
jobs:
  myjob:
    parameters:
      preinstall-foo:
        type: boolean
        default: false
    machine: true
    steps:
      - run: echo "preinstall is << parameters.preinstall-foo >>"
      - when:
          condition: << parameters.preinstall-foo >>
          steps:
            - run: echo "preinstall"
      - unless:
          condition: << parameters.preinstall-foo >>
          steps:
            - run: echo "don't preinstall"

workflows:
  workflow:
    jobs:
      - myjob:
          preinstall-foo: false
      - myjob:
          preinstall-foo: true

Note: Conditional steps are available in configuration version 2.1 and later.