This guide was originally posted in Japanese here.
Interested in contributing a post to this blog? Reach out to our team at blog at circleci.com

Recently, CircleCI 2.0 went into open beta. Since CircleCI 1.0 adopted LXC as its base container, it hasn’t yet been possible to use Docker versions 1.11 or above. Luckily, there’s no such restriction in CircleCI 2.0 since it supports Docker natively.

In CircleCI 1.0 projects, you had to choose from two choices: Ubuntu 12.04 (Precise) or Ubuntu 14.04 (Trusty). In CircleCI 2.0, users can freely combine arbitrary images like LEGO blocks to create the desired CI container environments.

However, even in CircleCI 1.0, you could build, push and deploy the application as a Docker image. CircleCI 2.0 builds as a Docker image using Docker in a Docker container; how is that even possible?!

Fortunately, CircleCI 2.0 also answers this Docker-in-Docker problem. In this post, I’ll briefly describe how to build a Docker image in CircleCI 2.0, including the image layer cache.

TL; DR

In your project’s root directory, create a config.yml in a directory called .circleci:

version: 2
jobs:
  build:
    working_directory: /app
    docker:
      - image: docker:17.05.0-ce-git
    steps:
      - checkout
      - setup_remote_docker
      - run:
          name: Install dependencies
          command: |
            apk add --no-cache \
              py-pip=9.0.0-r1
            pip install \
              docker-compose==1.12.0 \
              awscli==1.11.76
      - restore_cache:
          keys:
            - v1-{{ .Branch }}
          paths:
            - /caches/app.tar
      - run:
          name: Load Docker image layer cache
          command: |
            set +o pipefail
            docker load -i /caches/app.tar | true
      - run:
          name: Build application Docker image
          command: |
            docker build --cache-from=app -t app .
      - run:
          name: Save Docker image layer cache
          command: |
            mkdir -p /caches
            docker save -o /caches/app.tar app
      - save_cache:
          key: v1-{{ .Branch }}-{{ epoch }}
          paths:
            - /caches/app.tar
      - run:
          name: Run tests
          command: |
            docker-compose -f ./docker-compose.test.yml up
      - deploy:
          name: Push application Docker image
          command: |
            if [ "${CIRCLE_BRANCH}" == "master" ]; then
              login="$(aws ecr get-login)"
              ${login}
              docker tag app "${ECR_ENDPOINT}/app:${CIRCLE_SHA1}"
              docker push "${ECR_ENDPOINT}/app:${CIRCLE_SHA1}"
            fi

Depending on the project, some of these details may change. I prepared a sample project that can use this config file.

The project is written in node.js using Express and simply returns “Hello World”, tested with Jest and supertest. To push the Docker image (ECR) to the Amazon EC2 Container Registry, we use a Terraform script.

In the following breakdown of each section, I’ll explain how I do this:

Top-level structure

version: 2
jobs:
  build:
    working_directory: /app
    docker:
      ...
    steps:
      ...

Full details are given in the official documentation, but the most noteworthy difference is this:

In CircleCI 1.0, the fixed execution section can be freely defined by the user as a job; the build job is the only job that CircleCI will execute automatically. In CircleCI 2.0, we can freely define the steps of each job, including repository checkout and cache-related processing.

In other words, we’ve gained greater freedom, not only for the CI container environment but also for the steps.

Docker executor

docker:
  - image: docker:17.05.0-ce-git

In this section, we define the CI environment mentioned earlier. This environment is where our steps will be executed.

What we want is a Docker image that installs Docker and has Git. These requirements are satisfied by using docker:17.05.0-ce-git, which is an offical Docker image.

When an image has the suffix “-git”, it means Git is pre-installed. By using this image, you ensure you’re always using the latest Docker client.

Checkout

steps:
  - checkout

The first step, checkout, is a special step to check out the source code; this will be downloaded to the directory specified by working_directory.

Setup remote Docker

  - setup_remote_docker

This step helps you avoid the Docker-in-Docker problem. In fact, we’re setting up an environment that is isolated from the CI (or primary) container, then using the remote host’s Docker Engine.

Install required libraries

  - run:
    name: Install dependencies
      command: |
        apk add --no-cache \
          py-pip=9.0.0-r1
        pip install \
          docker-compose==1.12.0 \
          awscli==1.11.76

Here, we install Python, pip, Docker Compose, and the AWS CLI.

In real projects, I recommend installing these dependencies inside your image in advance.

Build and cache Docker images

  - restore_cache:
    keys:
      - v1-{{ .Branch }}
    paths:
      - /caches/app.tar
  - run:
    name: Load Docker image layer cache
    command: |
      set +o pipefail
      docker load -i /caches/app.tar | true
  - run:
    name: Build application Docker image
    command: |
      docker build --cache-from=app -t app .
  - run:
    name: Save Docker image layer cache
    command: |
      mkdir -p /caches
      docker save -o /caches/app.tar app
  - save_cache:
    key: v1-{{ .Branch }}-{{ epoch }}
    paths:
      - /caches/app.tar

This is the heart of this post. Basically, we’re doing the following:

When there’s a cache suffixed with v1-{{ <branch name> }}, CircleCI will restore your directory to /caches/app.tar. app.tar is the Docker image file from the previous build.
When /caches/app.tar exists, Docker will load it, allowing us to reuse images from previous builds.
When you build a Docker image, you’ll need to specify --cache-from=<image name>.
We’ll save the Docker image we built in /caches/app.tar.
Finally, we cache /caches/app.tar so we can reuse it in the next build. We use v1-{{ <branch name> }}-{{ <Unix epoch time> }} as the cache key.

The reason we have to do all this is because the remote Docker engine doesn’t do layer caching by default. Although there’s a function to perform this layer caching, we’d have to ask CircleCI Support to enable the caching feature in the 2.0 open beta. It’s also possible that this might become a paid feature in the official release.

You can refer to the sample project’s build history to see how much speed improvement can actually be obtained by caching the image layer. For example, build #12 has a cache and build #13 builds with no cache. In this example, we only saw a speed increase of around 22 seconds.

Run tests

  - run:
    name: Run tests
      command: |
        docker-compose -f ./docker-compose.test.yml up

I’m running the tests with Docker Compose. In this sample project, the tests are only run in the application container. If you need a database container, it’s easy to set up using Docker Compose.

Push Docker image

  - deploy:
    name: Push application Docker image
      command: |
        if [ "$ {CIRCLE_BRANCH}" == "master" ]; then
          login="$(aws ecr get-login)"
          ${login}
          docker tag app "${ECR_ENDPOINT}/app:${CIRCLE_SHA1}"
          docker push "${ECR_ENDPOINT}/app:${CIRCLE_SHA1}"
        fi

Only when the branch is master will the Docker images be built and pushed to the ECR repository. We get our login information from the AWS CLI we installed earlier. In a real project, you’d normally deploy using ecs-deploy after pushing the image.

In conclusion

CircleCI 2.0 allows you to use the remote Docker engine to build Docker images. Even though the improvement in build speed was minor in my case, I still learned that caching of the Docker image layer could be done.

If you encounter problems, both the official docs and the community forums should be helpful.

Naoto Yokoyama is a freelance full stack engineer. You can get in touch with them on Twitter, GitHub, or their blog.

Engineering