layout: post date: ‘2021-04-30 17:28 -0700’ published: true author: circleci title: Building a Docker image on CircleCI description: >- Building a Docker image on CircleCI html_title: >- Building a Docker image on CircleCI summary: >- Building a Docker image on CircleCI tags:

• engineering featured: false

In CircleCI, developers can freely combine arbitrary images like LEGO blocks to create their preferred CI container environments. For example, CircleCI supports Docker natively. You can build, push and deploy the application as a Docker image. CircleCI builds as a Docker image using Docker in a Docker container,

In this post, I will briefly describe how to build a Docker image in CircleCI, including the image layer cache.

TL; DR

In your project’s root directory, create a config.yml in a directory called .circleci:

```version: 2 jobs: build: working_directory: /app docker: - image: docker:17.05.0-ce-git steps: - checkout - setup_remote_docker - run: name: Install dependencies command: | apk add –no-cache
py-pip=9.0.0-r1 pip install
docker-compose==1.12.0
awscli==1.11.76 - restore_cache: keys: - v1-{{ .Branch }} paths: - /caches/app.tar - run: name: Load Docker image layer cache command: | set +o pipefail docker load -i /caches/app.tar | true - run: name: Build application Docker image command: | docker build –cache-from=app -t app . - run: name: Save Docker image layer cache command: | mkdir -p /caches docker save -o /caches/app.tar app - save_cache: key: v1-{{ .Branch }}-{{ epoch }} paths: - /caches/app.tar - run: name: Run tests command: | docker-compose -f ./docker-compose.test.yml up - deploy: name: Push application Docker image command: | if [ “${CIRCLE_BRANCH}” == “master” ]; then login=”$(aws ecr get-login)” ${login} docker tag app “${ECR_ENDPOINT}/app:${CIRCLE_SHA1}” docker push “${ECR_ENDPOINT}/app:${CIRCLE_SHA1}” fi

Depending on the project, some of the details may change. If you want to try these steps for yourself, I prepared a [sample project](https://github.com/builtinnya/circleci-2.0-beta-docker-example) that you can use this config file with.

The project is written in node.js using [Express](https://expressjs.com/) and simply returns "Hello World", tested with [Jest](https://facebook.github.io/jest/) and [supertest](https://github.com/visionmedia/supertest). To push the Docker image (ECR) to the [Amazon EC2 Container Registry](https://aws.amazon.com/jp/ecr/), I use a [Terraform](https://www.terraform.io/) script.

Following each section, I will provide a breakdown of how it works.

## Top-level structure

```version: 2
jobs:
  build:
    working_directory: /app
    docker:
      ...
    steps:
      ...

In CircleCI, we can freely define the steps of each job, including repository checkout and cache-related processing. This gives us great freedom, not only for the CI container environment but also for the steps. You can find all the details in the official documentation.

Docker executor

docker:
  - image: docker:17.05.0-ce-git

In this section, we define the CI environment mentioned earlier. This environment is where our steps will be executed.

What we want is a Docker image that installs Docker and has Git. These requirements are satisfied by using docker:17.05.0-ce-git, which is an offical Docker image.

When an image has the suffix “-git”, it means Git is pre-installed. By using this image, you ensure you’re always using the latest Docker client.

Checkout

  - checkout

The first step, checkout, is a special step to check out the source code; this will be downloaded to the directory specified by working_directory.

Setup remote Docker

``` - setup_remote_docker

This step helps you avoid the Docker-in-Docker problem. In fact, we're setting up an [environment that is isolated](https://circleci.com/docs/2.0/building-docker-images/){:target="_blank" rel="noreferrer noopener"} from the CI (or _primary_) container, then using the remote host's Docker Engine.

## Install required libraries

``` - run:
    name: Install dependencies
    command: |
      apk add --no-cache \
        py-pip=9.0.0-r1
      pip install \
        docker-compose==1.12.0 \
        awscli==1.11.76

Here, we install Python, pip, Docker Compose, and the AWS CLI.

In real projects, I recommend installing these dependencies inside your image in advance.

Build and cache Docker images

``` - restore_cache: keys: - v1-{{ .Branch }} paths: - /caches/app.tar

• run: name: Load Docker image layer cache command: | set +o pipefail docker load -i /caches/app.tar | true
• run: name: Build application Docker image command: | docker build –cache-from=app -t app .
• run: name: Save Docker image layer cache command: | mkdir -p /caches docker save -o /caches/app.tar app
• save_cache: key: v1-{{ .Branch }}-{{ epoch }} paths:
  • /caches/app.tar ```

This is the heart of this post. Basically, we’re doing the following:

1. When there’s a cache suffixed with v1-{{ <branch name> }}, CircleCI will restore your directory to /caches/app.tar. app.tar is the Docker image file from the previous build.

2. When /caches/app.tar exists, Docker will load it, allowing us to reuse images from previous builds.

3. When you build a Docker image, you’ll need to specify --cache-from=<image name>.

4. We’ll save the Docker image we built in /caches/app.tar.

5. Finally, we cache /caches/app.tar so we can reuse it in the next build. We use v1-{{ <branch name> }}-{{ <Unix epoch time> }} as the cache key.

The reason we have to do all this is because the remote Docker engine doesn’t do layer caching by default. Although there’s a function to perform this layer caching, we’d have to ask CircleCI Support to enable the caching feature in the 2.0 open beta. It’s also possible that this might become a paid feature in the official release.

You can refer to the sample project’s build history to see how much speed improvement can actually be obtained by caching the image layer. For example, build #12 has a cache and build #13 builds with no cache. In this example, we saw a speed increase of around 22 seconds.

Run tests

``` - run: name: Run tests command: | docker-compose -f ./docker-compose.test.yml up

For this project I am running the tests with Docker Compose, and the tests are run only in the application container. If you need a database container, it is easy to set one up using Docker Compose.

## Push Docker image

```  - deploy:
    name: Push application Docker image
    command: |
      if [ "$ {CIRCLE_BRANCH}" == "master" ]; then
        login="$(aws ecr get-login)"
        ${login}
        docker tag app "${ECR_ENDPOINT}/app:${CIRCLE_SHA1}"
        docker push "${ECR_ENDPOINT}/app:${CIRCLE_SHA1}"
      fi

The Docker images are built and pushed to the ECR repository only for the main branch (in this case, called “master”). We get our login information from the AWS CLI we installed earlier. In a real project, you would normally deploy using ecs-deploy after pushing the image.

In conclusion

CircleCI allows you to use the remote Docker engine to build Docker images. Even though the improvement in build speed was minor in my case, I still learned that caching of the Docker image layer could be done. Depending on the project, build speed improvements could be much greater.

Read more:

• CircleCI and Docker: what you need to know
• How to build a CI/CD pipeline with Docker
• Using Docker effectively on CircleCI

Naoto Yokoyama is a freelance full stack engineer. You can get in touch with them on Twitter, GitHub, or their blog.