Start Building for Free
CircleCI.comAcademyBlogCommunitySupport

Optimization reference

5 days ago3 min read
Cloud
Server v4+
On This Page

This page provides an overview of the various ways you can optimize your CircleCI configuration. Each optimization method is described briefly, along with possible use cases.

Optimize your data usage

Custom storage controls

The CircleCI web app provides controls to customize the storage retention period for workspaces, caches, and artifacts. You can find these settings by navigating to Plan  Usage Controls. By default, the storage periods are set to the maximum: 30 days for artifacts, and 15 days for caches and workspaces.

See the Persisting Data page for more information on custom storage settings.

Workspaces

Workspaces are used to pass along data that is unique to a run and is needed for downstream jobs. A job that runs early in your workflow might fetch data and then make it available later for jobs that run later in that workflow.

To persist data from a job and make it available to downstream jobs via the attach_workspace key, configure the job to use the persist_to_workspace key. Files and directories named in the paths: property of persist_to_workspace will be uploaded to the workflow’s temporary workspace relative to the directory specified with the root key. The files and directories are then uploaded and made available for subsequent jobs (and re-runs of the workflow) to use.

Reduce build time

Docker image choice

Choosing the right Docker image for your project can have huge impact on build time. For example, choosing a basic language image means dependencies and tools need to be downloaded each time your pipeline is run, whereas, if you choose or build an image that has these dependencies and tools already installed, this time will be saved for each build run. When configuring your projects and specifying images, consider the following options:

Docker layer caching

Docker layer caching is a feature that can help to reduce the build time of a Docker image in your build. DLC is useful if you find yourself frequently building Docker images as a regular part of your CI/CD process.

DLC is similar to caching dependencies, in that it saves the image layers that you build within your job, making them available on subsequent builds.

Caching dependencies

Caching should be one of the first things you consider when trying to optimize your jobs. If a job fetches data at any point, it is likely that you can make use of caching. A common example is the use of a package/dependency manager. If your project uses Yarn, Bundler, or Pip, for example, the dependencies downloaded during a job can be cached for later use rather than being re-downloaded on every build.

Workflows

Workflows provide a means to define a collection of jobs and their run order. If at any point in your configuration you see a step where two jobs could happily run independent of one another, workflows may be helpful. Workflows also provide several other features to augment and improve your CI/CD configuration. Read more about workflows on the Workflow page.

Parallelism

If your project has a large test suite, you can configure your build to use parallelism together with either CircleCI’s test splitting functionality, or a third party application or library to split your tests across multiple machines. CircleCI supports automatic test allocation across machines/containers on a file-basis. You can also manually customize how tests are allocated.

Rerun failed tests

Use the rerun failed tests feature to only rerun a subset of tests when a transient test failure arises, rather than rerunning the entire test suite.

Resource class

Using resource_class, it is possible to specify CPU and RAM resources for each job. For a full list of available resource class options for CircleCI cloud see the configuration reference. For an equivalent list for CircleCI server installations, contact your system administrator.

Examples of how to set a resource class:

Optimize your configuration files

Dynamic configuration

Use dynamic configuration to generate CircleCI config files dynamically, depending on specific pipeline values or file paths. Dynamic config allows you to:

  • Execute conditional workflows/commands.

  • Pass pipeline parameter values and/or generate additional configuration.

  • Trigger separate config.yml configurations, which exist outside the default parent .circleci/ directory.

Read more about dynamic configuration on the Dynamic configuration overview page.

Orbs

Orbs are reusable packages of parameterizable configuration that can be used in any project. Use orbs to:

  • Simplify configuration (.circleci/_config.yml)

  • Automate repeated processes

  • Accelerate project setup

  • Simplify integration with third-party tools

Read more about orbs on the Orbs overview page.

See also


Suggest an edit to this page

Make a contribution
Learn how to contribute