cmd+/

Docker layer caching overview

Cloud Server v4+

Use Docker layer caching (DLC) to reduce Docker image build times on CircleCI. DLC is available on all CircleCI plans.

Introduction

Docker layer caching (DLC) is beneficial if building Docker images is a regular part of your CI/CD process. DLC saves Docker image layers created within your jobs and reuses them in future builds.

DLC caches the individual layers of any Docker images built during your CircleCI jobs. On subsequent job runs, CircleCI reuses unchanged image layers instead of rebuilding the entire image every time. In short, the less your Dockerfiles change from commit to commit, the less time your image-building jobs take to run.

You can use DLC with both the machine executor and in a Remote Docker Environment (setup_remote_docker).

DLC costs 200 credits per job run. For more information about DLC charges, see the FAQ.

DLC has a couple of temporary, known issues. See Known Issues before enabling it on gen3 resource classes or newer Docker versions.

Quickstart

Remote Docker environment

To use DLC with the Docker execution environment, configure your job to run in a Remote Docker Environment. To do this, add docker_layer_caching: true under the setup_remote_docker key in your .circleci/config.yml file:

version: 2.1

jobs:
 build:
    docker:
      - image: cimg/base:2023.04

    steps:
      - checkout
      - setup_remote_docker:
          docker_layer_caching: true

Machine executor

You can use DLC when building Docker images with the machine executor. Use DLC with the machine executor by adding docker_layer_caching: true below your machine key:

version: 2.1

jobs:
 build:
    machine:
      image: ubuntu-2404:current  # any available image
      docker_layer_caching: true    # default - false

    steps:
      - checkout

Known issues

Gen3 resource classes causing infrastructure failures

We are investigating reports that enabling DLC (docker_layer_caching: true) on gen3 resource classes (*.gen3) can cause infrastructure failures. This incompatibility is a temporary, known issue, not a permanent limitation of DLC or gen3. Avoid enabling DLC on gen3 resource classes for now, whether you are using the machine executor or a Remote Docker Environment.

Docker 29 compatibility

DLC is not currently compatible with Docker 29 or later. This incompatibility is a temporary, known issue, and we are working on a fix. Avoid docker29 if you use DLC. See Specify a Docker Version for Remote Docker for how to pin a version.

Limitations

Building Docker images

DLC is only useful when creating your own Docker image with docker build, docker compose, or similar Docker commands. It does not decrease the wall clock time that all builds take to spin up the initial environment.

version: 2.1

orbs:
  browser-tools: circleci/browser-tools@1.2.3

jobs:
 build:
    docker:
      - image: cimg/node:17.2-browsers

    steps:
      - checkout
      - setup_remote_docker:
          docker_layer_caching: true # DLC will explicitly cache layers here and try to avoid rebuilding.
      - run: docker build .

DLC has no effect on Docker images used as build containers. That is, you specify the containers that run your jobs with the image key when using the docker executor, and they appear in the Spin up Environment step on your jobs pages.

Buildx builder instances

When using Buildx builder instances with DLC, name your builders. Doing so ensures CircleCI can detect and preserve the Docker volumes for subsequent job runs. If you do not name your builders, each job generates different, randomly generated names, and CircleCI automatically cleans up the resulting volumes.

DLC cannot maintain Docker build artifacts stored in Docker volumes managed by BuildKit inside Buildx builder instances, but Buildx can still support these artifacts. DLC is not able to prune these images/build cache, but Buildx builders do have some in-built pruning. For more information, see the Docker docs.

How DLC works

DLC caches your Docker image layers within the container/virtual machine used to run your job.

For example, if building a Docker image takes over two minutes on the first run and the Dockerfile does not change, subsequent builds happen instantly.

When none of the layers in the image change between job runs, DLC pulls the layers from the cache and reuses those instead of rebuilding the entire image.

If part of the Dockerfile changes, a subsequent run of the exact same job with the modified Dockerfile may still finish faster than rebuilding the entire image. This speed optimization happens because CircleCI can still use the cache for the initial steps that did not change in the Dockerfile. The steps that follow the change must be rerun because the Dockerfile change invalidates the cache for those layers.

If you change something in your Dockerfile, CircleCI invalidates all later steps from the point of the change and rebuilds those layers. When some steps remain the same (the steps before the one you removed), CircleCI can reuse those steps, so the build is still faster than rebuilding the entire image.

At the beginning of each job that uses DLC, there is a DLC set-up step. You are not charged for this step. At the end of each job that uses DLC, CircleCI uploads the cache asynchronously, which does not prevent the workflow from continuing to progress. This means that jobs within the same workflow generally do not access a cache uploaded from an upstream job. You are not charged for this DLC teardown step.

Parallelism and DLC

DLC operates in the same way for jobs that use parallelism. If you configure a machine job using DLC with parallelism: 2, two jobs run in parallel. Each virtual machine has a separate DLC cache, and CircleCI uses the last-saved cache for the next build.

Purge DLC

If jobs that use DLC continuously fail, this may be due to a corrupted cache. Purging the DLC removes previously stored Docker image layers so that future builds do not reuse those cached layers. This forces CircleCI to rebuild Docker images from scratch, rather than leveraging layers from earlier builds to speed up the process.

You can purge DLC using the CircleCI web app or the CLI.

Web app

To purge DLC from the CircleCI web app:

In the CircleCI web app, select your org from the org cards on your user homepage.
Select Projects from the sidebar and locate your project from the list. You can use the search to help.
Select the ellipsis next to your project and select Project Settings.

You can also access project settings from each project overview page using the Settings button.
Select Docker Layer Caching.
Select Delete Cache Contents.

CLI

Use the CircleCI CLI dlc purge sub-command to purge DLC for a project:

circleci dlc purge <vcs-type> <org-id> <project-id>

The full <vcs-type> <org-id> <project-id> section of the subcommand is your project slug. You can find and copy the project slug from your project settings:

Screenshot showing where to find the project slug

Figure 1. The project slug available in project settings

For full details on the dlc purge sub-command, refer to the CLI reference.

Deprecated keys

DLC was previously enabled via the reusable: true key. We deprecated the reusable key in favor of the docker_layer_caching key.

We also deprecated the exclusive: true option, and CircleCI now treats all remote Docker VMs as exclusive. This means that when using DLC, jobs get an exclusive remote Docker environment that other jobs cannot access.

[path]

[title]