From config disaster to config build-faster

The customer engineering team at CircleCI helps users optimize how their configuration files are set up everyday - they find the most useful features for your projects, balancing both your time and your credit consumption. But teams don’t always have time to work with an expert to optimize their config. That’s why, after performing many config reviews for 20+ of our enterprise-level customers, our customer engineering team has put together a config optimization guide with their best tips and recommendations to make it easier for teams to optimize their config.

In this post, we’ll cover six different ways to optimize your config file. We’ll walk through best practices for picking the right executor, parallelizing jobs, caching, using workspaces, secrets management, and using orbs in your config.

Selecting the right executor

Many CI pipelines would benefit from choosing one of our 2020 fleet of lightning-fast Docker convenience images. Running within a docker container using the docker yaml key will provide the basics at some of the fastest speeds.

We publish these to the Docker Hub cimg profile. If your application needs other tools, consider running a custom Docker image. Here is an example using one of the older, bulkier images pinned to a certain version of Node. This executor is defined under each job by specifying a Docker image, such as in this test job:

test:
    docker:
      - image: circleci/node:9.9.

With the next-gen CircleCI Node image, you can shed layers, giving a faster build. Updating to the next-gen executor is as simple as updating the image name.

The current config builds and tests in Node 9.9.0, but we would like it to be built using the latest version of Node. To do this, we replace the image used for the execution container with one of our next-gen images as follows:

docker:
  - image: cimg/node:latest

If you are interested in testing across multiple environments, we also have the ability to set matrix jobs via the Node Orb. This allows you to specify different versions of Node to test as well on top of the base Node Docker layers.

Best practices for parallelism

Configuring your job to run across multiple containers in parallel speeds up your build. For example, if you have a long running test suite with hundreds of independent tests, consider spreading these tests across executors to run simultaneously. A truly optimized config means leveraging parallelism wisely. You should carefully consider how many parallel executors you have running and whether the time-savings of splitting up tasks is worth the spin-up time of multiple containers. Also ensure that you’re splitting your tests correctly across these executors.
Consider the example below. The primary activity in this test job is running the tests with the npm run test command:

test:
 …
 parallelism: 10
 steps:
    ...
     - run: CI=true npm run test

While using parallelism is a step in the right direction, this is not written optimally. This command will simply run the same exact tests on all 10 containers. To allocate the tests across multiple containers, this config needs to use the circleci tests split command from the CircleCI CLI. Tests can be automatically allocated across these containers when split by filename, classname, or timing data. Splitting by timing data is ideal for parallelization, as it spreads the tests to run evenly across the containers, so there are no faster-running containers waiting idly for a long-running test container.

Finally, consider whether this is the correct level of parallelism for this test suite. If spinning up the environment takes about 30 seconds, but testing only takes 30 seconds in each container, it may be worth considering lowering the parallelism so less time is spent setting up across all of the job runs. There is no golden ratio of test runtime to spin-up time, but it should be considered for an optimal build. Here is what an optimized config looks like when optimized to split tests by filename and timings and to run more tests in a given container:

test:
 …
 parallelism: 5
 steps:
    ...
     - run: |
            TESTFILES=$(circleci tests glob "test/**/*.test.js" | circleci tests split --split-by=timings)
            CI=true npm run test $TESTFILES

Best practices for caching

Speed up your builds with caching. This allows you to reuse data from time-consuming fetch operations. The example below uses caching to restore its npm dependencies from previous job runs. Because the npm dependencies are cached, the npm install step will only need to download new dependencies described in the package.json file. This dependency caching, which is often used with package dependency managers like npm, Yarn, Bundler, or pip, relies on two special steps specified as restore_cache and save_cache. Let’s look at how these cache steps are used in the test job:

test:
  ...
  steps:
    …
    - restore_cache:
        keys:
          - v1-deps-{{ checksum "package-lock.json" }}
    - run: npm install
    - save_cache:
        key: v1-deps-{{ checksum "package-lock.json" }}
        paths:
          - node_modules

Notice that both the restore_cache and save_cache steps use keys. This key is a unique identifier for locating your cache. The save_cache step specifies which directories to cache under this key. In this case, we are saving the node_modules directory, so that these Node dependencies can be used in later jobs. The restore_cache step uses this key to find the cache to restore to the job. In this case, the key is a string with a version identifying the cache and an interpolated hash of the dependency manifest file written as checksum “package-lock.json”.

While this is a standard pattern of restoring and saving a cache, this can be optimized with the use of fallback keys. Fallback keys allow you to identify a set of possible caches to increase the likelihood of a cache hit. For example, if a single package is added to this application’s package.json, the string generated by checksum will change, and the entire cache will be missed. However, adding a fallback key with a broader set of possible key matches can identify other usable caches. Here is an example of what this cache restoration would look like with an added fallback key:

test:
  ...
  steps:
    …
    - restore_cache:
        keys:
          - v1-deps-{{ checksum "package-lock.json" }}
          - v1-deps-
    

Notice that we just added another element to the list of keys. Let’s go back to the scenario where a single package changed in our package.json. In this case, the first key would result in a cache miss. However, the second key allows for the cache that was previously saved within the old package.json file to be restored. The dependency installation step, npm install, would then only have to fetch the changed packages, as opposed to unnecessary and expensive fetch operations for all of the packages. Visit our docs to read more about fallback keys and partial cache restoration.

Selectively persisting to workspaces

Downstream jobs may need access to data that was generated in a prior job. Workspaces allow you to store files for the entire life of the workflow. To illustrate this, let’s look at the config below. The build job builds a Node application. The next job in the workflow deploys the application. This config persists the entire working directory to the workspace in build, then attaches the directory in deploy, so deploy has access to the built application:

  build:
    ...
    steps:
      ...
      - run: npm run build
      - persist_to_workspace:
          root: .
          paths:
            - '*'
  deploy:
    ...
    steps:
       ....
        - attach_workspace:
            at: .

This works, as the application directory created in the build job would be accessible to the deploy job, but it is not ideal. Workspaces essentially just create tarballs and store them in a blob store, and attaching a workspace requires downloading and unpacking these tarballs. This can be a time-consuming process. It would be faster to selectively persist the files that your later jobs need. In this case, let’s say the npm run build step produces a build directory that can be compressed then stored in the workspace for deployment. You can see what this might look like in the optimized version of this config below:

  build:
    ...
    steps:
      ...
      - run: npm run build
      - run: mkdir tmp && zip -r tmp/build.zip build
      - persist_to_workspace:
          root: .
          paths:
            - 'tmp'

  deploy:
    ...
    steps:
       ....
        - attach_workspace:
            at: .

The tmp directory with the build artifact will now be mounted to the working directory of the project. Rather than uploading and downloading the entire working directory as done in the unoptimized config, this config selectively stores the compressed, built application to save on time spent archiving, uploading, and downloading the workspace. The compressed file can be stored in a temporary directory in the workspace. Any downstream jobs with the workspace attached to them will now have access to this zip file. You can learn more in this deep dive into CircleCI workspaces.

Best practices for secrets management

You do not want to check your secrets into version control, and secrets should never be written in plain text in your config. CircleCI provides you with access to contexts. These allow you to secure and share environment variables across projects in your organization. Contexts are essentially a secret store where you can set environment variables as name/value pairs that are injected at runtime. To understand this better, let’s look at the unsecure config below. This config includes a deploy job that is defined with AWS secrets written in plain-text:

deploy:
 …
 steps:
    ...
     - run: 
            name: Configure AWS Access Key ID
            command: |
              aws configure set aws_access_key_id K4GMW195WJKGCWVLGPZG --profile default
        - run: 
            name: Configure AWS Secret Access Key
            command: |
              aws configure set aws_secret_access_key ka1rt3Rff8beXPTEmvVF4j4DZX3gbi6Y521W1oAt --profile default

Note: These are dummy credentials used for demonstration purposes only.

This text is visible to all of the developers with access to your project on CircleCI. These secrets should be stored as environment variables in a context instead. Add the secret key and access ID to a context titled aws_secrets as key/value pairs, which can be accessed as environment variables. This context can then be applied to the job in the workflow. The secure version of this config would look like this:

deploy:
 …
 steps:
    ...
     - run: 
            name: Configure AWS Access Key ID
            command: |
              aws configure set aws_access_key_id ${AWS_ACCESS_KEY_ID} --profile default
        - run: 
            name: Configure AWS Secret Access Key
            command: |
              aws configure set aws_secret_access_key ${AWS_SECRET_ACCESS_KEY} --profile default

workflows:
  test-build-deploy:
     ...
      - deploy:
          context: aws_secrets
          requires:
            - build

Notice that the secrets have gone from plain-text to an environment variable and the context is applied to the job in the workflow. For additional security, we employ secret masking to prevent users from accidentally printing the value of the secret.

Orbs and reusable config elements

So you’ve chosen the right executor for your build, you’re splitting your tests appropriately, and you’re persisting to the workspace to avoid duplicating your work. Now you have to do that for all of your other projects. What a pain, am I right? If only there was a way to reuse shared elements of your config file between multiple builds. Well, good news!

Circle CI provides a feature known as orbs which allow you to define configuration elements in a central location and reuse them in multiple projects quickly and easily. Not just that, but you can actually pass parameters into orbs, so you can craft a single orb that does multiple different things in different projects depending on the parameters you pass into it.

Using our 2.1 config version, you can also define reusable elements of your configuration to re-use in multiple jobs in the same pipeline, from simple job steps to reusing entire executors. You can also pass parameters into these reusable elements. This is useful for when you need to reuse multiple elements of a config file across multiple different parts of your pipeline.

What do orbs look like in practice? Well, here’s an example of a deployment to an S3 bucket, written entirely in the config file without the use of our AWS S3 deployment orb:

- deploy:
    name: S3 Sync
    command: |+
      aws s3 sync \
        build s3://my-s3-bucket-name/my-application --delete \
        --acl public-read \
      --cache-control "max-age=86400"

That’ll get the job done. But here’s what it looks like with the use of the S3 orb:

- aws-s3/sync:
     from: bucket
     to: ‘s3://my-s3-bucket-name/my-application’
     arguments: |
       --acl public-read \
       --cache-control "max-age=86400"

You don’t need to declare a separate deploy stage for your S3 deployment. You can simply invoke the S3 sync from the orb as a step in your config file. Additionally, note that a lot of the same information is still included, but it is now represented as parameters that are passed into the orb instead of as a script in the configuration file. Not only is this more compact, but it also makes it very easy to make changes to your S3 deployment as needed by adding, removing, or changing arguments. Plus, it’s a bit easier to grasp at a glance and scale across multiple projects by updating just the orb. The most important takeaway from all the tips mentioned here is that D.R.Y - ness isn’t just an esthetic thing. With orbs, the ability to replicate across projects is golden. Happy orb-optimizing!

Configuration review

As part of our premium support packages, we offer a configuration review premium service on our Gold and Platinum support plans. With the configuration review service, a CircleCI DevOps customer engineer will help your team build a new config or review an existing one, so you can save time in your build cycle. We’ll review your needs and provide recommendations to get the most out of CircleCI’s features.

If you’re interested in learning more about the available premium services and Support Plans, please contact us at cs@circleci.com.


This post would not have been possible without the combined efforts of the entire customer engineering team: Anna Calinawan, Johanna Griffin, and Grant MacGillivray.