This guide was originally posted in Japanese here. Interested in contributing a post to this blog? Reach out to our team at blog at circleci.com
Update: Beginning July 11, 2017, CircleCI 2.0 is out of beta and publicly available. Read all about it.
Recently, CircleCI 2.0 went into open beta. Since CircleCI 1.0 adopted LXC as its base container, it hasn’t yet been possible to use Docker versions 1.11 or above. Luckily, there’s no such restriction in CircleCI 2.0 since it supports Docker natively.
In CircleCI 1.0 projects, you had to choose from two choices: Ubuntu 12.04 (Precise) or Ubuntu 14.04 (Trusty). In CircleCI 2.0, users can freely combine arbitrary images like LEGO blocks to create the desired CI container environments.
However, even in CircleCI 1.0, you could build, push and deploy the application as a Docker image. CircleCI 2.0 builds as a Docker image using Docker in a Docker container; how is that even possible?!
Fortunately, CircleCI 2.0 also answers this Docker-in-Docker problem. In this post, I’ll briefly describe how to build a Docker image in CircleCI 2.0, including the image layer cache.
TL; DR
In your project’s root directory, create a config.yml in a directory called .circleci:
version: 2
jobs:
build:
working_directory: /app
docker:
- image: docker:17.05.0-ce-git
steps:
- checkout
- setup_remote_docker
- run:
name: Install dependencies
command: |
apk add --no-cache \
py-pip=9.0.0-r1
pip install \
docker-compose==1.12.0 \
awscli==1.11.76
- restore_cache:
keys:
- v1-{{ .Branch }}
paths:
- /caches/app.tar
- run:
name: Load Docker image layer cache
command: |
set +o pipefail
docker load -i /caches/app.tar | true
- run:
name: Build application Docker image
command: |
docker build --cache-from=app -t app .
- run:
name: Save Docker image layer cache
command: |
mkdir -p /caches
docker save -o /caches/app.tar app
- save_cache:
key: v1-{{ .Branch }}-{{ epoch }}
paths:
- /caches/app.tar
- run:
name: Run tests
command: |
docker-compose -f ./docker-compose.test.yml up
- deploy:
name: Push application Docker image
command: |
if [ "${CIRCLE_BRANCH}" == "master" ]; then
login="$(aws ecr get-login)"
${login}
docker tag app "${ECR_ENDPOINT}/app:${CIRCLE_SHA1}"
docker push "${ECR_ENDPOINT}/app:${CIRCLE_SHA1}"
fi
Depending on the project, some of these details may change. I prepared a sample project that can use this config file.
The project is written in node.js using Express and simply returns “Hello World”, tested with Jest and supertest. To push the Docker image (ECR) to the Amazon EC2 Container Registry, we use a Terraform script.
In the following breakdown of each section, I’ll explain how I do this:
Top-level structure
version: 2
jobs:
build:
working_directory: /app
docker:
...
steps:
...
Full details are given in the official documentation, but the most noteworthy difference is this:
In CircleCI 1.0, the fixed execution section can be freely defined by the user as a job; the build job is the only job that CircleCI will execute automatically. In CircleCI 2.0, we can freely define the steps of each job, including repository checkout and cache-related processing.
In other words, we’ve gained greater freedom, not only for the CI container environment but also for the steps.
Docker executor
docker:
- image: docker:17.05.0-ce-git
In this section, we define the CI environment mentioned earlier. This environment is where our steps will be executed.
What we want is a Docker image that installs Docker and has Git. These requirements are satisfied by using docker:17.05.0-ce-git, which is an offical Docker image.
When an image has the suffix “-git”, it means Git is pre-installed. By using this image, you ensure you’re always using the latest Docker client.
Checkout
steps:
- checkout
The first step, checkout, is a special step to check out the source code; this will be downloaded to the directory specified by working_directory.
Setup remote Docker
- setup_remote_docker
This step helps you avoid the Docker-in-Docker problem. In fact, we’re setting up an environment that is isolated from the CI (or primary) container, then using the remote host’s Docker Engine.
Install required libraries
- run:
name: Install dependencies
command: |
apk add --no-cache \
py-pip=9.0.0-r1
pip install \
docker-compose==1.12.0 \
awscli==1.11.76
Here, we install Python, pip, Docker Compose, and the AWS CLI.
In real projects, I recommend installing these dependencies inside your image in advance.
Build and cache Docker images
- restore_cache:
keys:
- v1-{{ .Branch }}
paths:
- /caches/app.tar
- run:
name: Load Docker image layer cache
command: |
set +o pipefail
docker load -i /caches/app.tar | true
- run:
name: Build application Docker image
command: |
docker build --cache-from=app -t app .
- run:
name: Save Docker image layer cache
command: |
mkdir -p /caches
docker save -o /caches/app.tar app
- save_cache:
key: v1-{{ .Branch }}-{{ epoch }}
paths:
- /caches/app.tar
This is the heart of this post. Basically, we’re doing the following:
-
When there’s a cache suffixed with
v1-{{ <branch name> }}, CircleCI will restore your directory to/caches/app.tar.app.taris the Docker image file from the previous build. -
When
/caches/app.tarexists, Docker will load it, allowing us to reuse images from previous builds. -
When you build a Docker image, you’ll need to specify
--cache-from=<image name>. -
We’ll save the Docker image we built in
/caches/app.tar. -
Finally, we cache
/caches/app.tarso we can reuse it in the next build. We usev1-{{ <branch name> }}-{{ <Unix epoch time> }}as the cache key.
The reason we have to do all this is because the remote Docker engine doesn’t do layer caching by default. Although there’s a function to perform this layer caching, we’d have to ask CircleCI Support to enable the caching feature in the 2.0 open beta. It’s also possible that this might become a paid feature in the official release.
You can refer to the sample project’s build history to see how much speed improvement can actually be obtained by caching the image layer. For example, build #12 has a cache and build #13 builds with no cache. In this example, we only saw a speed increase of around 22 seconds.
Run tests
- run:
name: Run tests
command: |
docker-compose -f ./docker-compose.test.yml up
I’m running the tests with Docker Compose. In this sample project, the tests are only run in the application container. If you need a database container, it’s easy to set up using Docker Compose.
Push Docker image
- deploy:
name: Push application Docker image
command: |
if [ "$ {CIRCLE_BRANCH}" == "master" ]; then
login="$(aws ecr get-login)"
${login}
docker tag app "${ECR_ENDPOINT}/app:${CIRCLE_SHA1}"
docker push "${ECR_ENDPOINT}/app:${CIRCLE_SHA1}"
fi
Only when the branch is master will the Docker images be built and pushed to the ECR repository. We get our login information from the AWS CLI we installed earlier. In a real project, you’d normally deploy using ecs-deploy after pushing the image.
In conclusion
CircleCI 2.0 allows you to use the remote Docker engine to build Docker images. Even though the improvement in build speed was minor in my case, I still learned that caching of the Docker image layer could be done.
If you encounter problems, both the official docs and the community forums should be helpful.
Read more:
- CircleCI and Docker: what you need to know
- How to build a CI/CD pipeline with Docker
- Using Docker effectively on CircleCI
Naoto Yokoyama is a freelance full stack engineer. You can get in touch with them on Twitter, GitHub, or their blog.
