Using custom-built Docker images

Cloud

Server v4+

Overview

CircleCI supports Docker, providing you with a powerful way to specify dependencies for your projects. If the CircleCI convenience images do not suit your needs, consider creating a custom Docker image for your jobs. There are two major benefits:

Faster job execution: Packaging your required tools into a custom image removes the need to install them for every job.
Cleaner configuration: Adding lengthy installation scripts to a custom image reduces the number of lines in your config.yml file.

When building Docker images, CircleCI does not preserve entrypoints by default. See Adding an Entrypoint for more details.

Create a custom image manually

The following sections provide a walkthrough of how to create a custom image manually. In most cases you will want to have a custom image for your primary container so that is the focus of this guide. You can also apply this knowledge to create images for secondary containers.

Prerequisites

A working Docker installation. For more details, see Docker’s Getting Started documentation.

1. Create a `Dockerfile`

To create a custom image, you must create a Dockerfile. This is a text document containing commands that Docker uses to assemble an image. Consider keeping your Dockerfile in your .circleci/images folder, as shown in this Docker demo project.

2. Choose and set a base image

Before you create a custom image, you must choose another image from which to extend the custom image. Docker Hub has official, pre-built images for most popular languages and frameworks. Given a particular language or framework, there are many image variants from which to choose. You can specify variants using Docker tags.

For example, if you want to use version 3.5 of the official Alpine image, the full image name is alpine:3.5.

In your Dockerfile, extend the base image by using the FROM instruction.

FROM golang:1.8.0

3. Install additional tools

To install any additional tools or execute other commands, use the RUN instruction.

RUN apt-get update && apt-get install -y netcat
RUN go get github.com/jstemmer/go-junit-report

Required tools for primary containers

In order to be used as a primary container on CircleCI, a custom Docker image must have the following tools installed:

Bash (most likely already installed or available via your package manager)
git
SSH
tar
Gzip
ca-certificates

Without these tools, some CircleCI services may not work.

If you do not install these tools with a package manager, you must use the ADD instruction instead of RUN (see below).

4. Add other files and directories

To add files and directories that are not present in package managers, use the ADD instruction.

ADD ./workdir/contacts /usr/bin/contacts
ADD ./db/migrations /migrations

5. Add an `entrypoint`

To run the container as an executable, use the ENTRYPOINT instruction. By default, CircleCI will ignore the entrypoint for a job’s primary container. To preserve the entrypoint even when the image is used for a primary container, use the LABEL instruction as shown below.

LABEL com.circleci.preserve-entrypoint=true

ENTRYPOINT contacts

Entrypoints should be commands that run forever without failing. If the entrypoint fails or terminates in the middle of a build, the build will also terminate. If you need to access logs or build status, consider using a background step instead of an entrypoint.

6. Build the image

After the required tools are specified in the Dockerfile you can build the image.

docker build <path-to-dockerfile>

You will see how all commands specified in you Dockerfile are executed. If there are any errors they will be displayed and you will need to fix them before continuing. If the build is successful you will have something like this at the end:

...
Successfully built e32703162dd4

Read more about the docker build command.

Congratulations, you have just built your first image! Now we need to store it somewhere to make it available for CircleCI.

7. Store image in a Docker registry

To allow CircleCI to use your custom image, store it in a public Docker Registry. The easiest mechanism is to create an account on Docker Hub, because Docker Hub allows you to store unlimited public images for free. If your organization is already using Docker Hub you can use your existing account.

To use an image with the CircleCI Docker Executor you must have a public repository. If you want to keep your image private, refer to the Using Docker Authenticated Pulls document for instructions.

The example uses Docker Hub, but you can use different registries if you prefer. Adapt the example based on the registry you are using.

8. Prepare the image for the registry

Log in to Docker Hub with your account and create a new repository on the add repository page. It is best practice to use a pattern similar to <project-name>-<container-name> for a repository name (for example, cci-demo-docker-primary).
Next, rebuild your image using your account and repository name:
```
docker build -t circleci/cci-demo-docker-primary:0.0.1 <path-to-dockerfile>
```
The -t key specifies the name and tag of the new image:
- circleci - our account in Docker Hub
- cci-demo-docker-primary - repository name
- 0.0.1 - tag (version) of the image. Always update the tag if you change something in a Dockerfile otherwise you might have unpredictable results.

9. Push the image to the registry

Push the image to Docker Hub:

$ docker login
$ docker push circleci/cci-demo-docker-primary:0.0.1

First, we use docker login to authenticate in Docker Hub. If you use a registry other than Docker Hub, refer to the related documentation about how to push images to that registry.

10. Use your image on CircleCI

After the image has been pushed, it is available for use it in your .circleci/config.yml:

version: 2.1
jobs:
  build:
    docker:
      - image: circleci/cci-demo-docker-primary:0.0.1

Using Docker? Authenticating Docker pulls from image registries is recommended when using the Docker execution environment. Authenticated pulls allow access to private Docker images, and may also grant higher rate limits, depending on your registry provider. For further information see Using Docker authenticated pulls.

If you have any questions, head over to our community forum.

Detailed custom Dockerfile example for Ruby

This section assumes you have already used Docker login locally.

This section demonstrates how to build a Ruby container to use on CircleCI.

The example starts with the Ruby 2.1 image. However, instead of using FROM ruby:2.1 as the base image it describes how the container is built. From the Ruby Docker Hub page, go to the 2.1/Dockerfile.

Notice the environment variables that are used to pull in the correct versions.

FROM buildpack-deps:jessie

# Skip installing gem documentation
RUN mkdir -p /usr/local/etc \
	&& { \
		echo 'install: --no-document'; \
		echo 'update: --no-document'; \
	} >> /usr/local/etc/gemrc

ENV RUBY_MAJOR 2.1
ENV RUBY_VERSION 2.1.10
ENV RUBY2_DOWNLOAD_SHA256 5be9f8d5d29d252cd7f969ab7550e31bbb001feb4a83532301c0dd3b5006e148
ENV RUBYGEMS_VERSION 2.6.10

# some of ruby's build scripts are written in ruby
#   we purge system ruby later to make sure our final image uses what we just built
RUN set -ex \
	\
	&& buildDeps=' \
		bison \
		libgdbm-dev \
		ruby \
	' \
	&& apt-get update \
	&& apt-get install -y --no-install-recommends $buildDeps \
	&& rm -rf /var/lib/apt/lists/* \
	\
	&& wget -O ruby.tar.xz "https://cache.ruby-lang.org/pub/ruby/${RUBY_MAJOR%-rc}/ruby-$RUBY_VERSION.tar.xz" \
	&& echo "$RUBY_DOWNLOAD_SHA256 *ruby.tar.xz" | sha256sum -c - \
	\
	&& mkdir -p /usr/src/ruby \
	&& tar -xJf ruby.tar.xz -C /usr/src/ruby --strip-components=1 \
	&& rm ruby.tar.xz \
	\
	&& cd /usr/src/ruby \
	\
# hack in "ENABLE_PATH_CHECK" disabling to suppress:
#   warning: Insecure world writable dir
	&& { \
		echo '#define ENABLE_PATH_CHECK 0'; \
		echo; \
		cat file.c; \
	} > file.c.new \
	&& mv file.c.new file.c \
	\
	&& autoconf \
	&& ./configure --disable-install-doc --enable-shared \
	&& make -j"$(nproc)" \
	&& make install \
	\
	&& apt-get purge -y --auto-remove $buildDeps \
	&& cd / \
	&& rm -r /usr/src/ruby \
	\
	&& gem update --system "$RUBYGEMS_VERSION"

ENV BUNDLER_VERSION 1.14.3

RUN gem install bundler --version "$BUNDLER_VERSION"

# install things globally, for great justice
# and don't create ".bundle" in all our apps
ENV GEM_HOME /usr/local/bundle
ENV BUNDLE_PATH="$GEM_HOME" \
	BUNDLE_BIN="$GEM_HOME/bin" \
	BUNDLE_SILENCE_ROOT_WARNING=1 \
	BUNDLE_APP_CONFIG="$GEM_HOME"
ENV PATH $BUNDLE_BIN:$PATH
RUN mkdir -p "$GEM_HOME" "$BUNDLE_BIN" \
	&& chmod 777 "$GEM_HOME" "$BUNDLE_BIN"

CMD [ "irb" ]

This will create a Ruby 2.1 image. Next, install node modules, awscli, and PostgreSQL 9.5 using the node:7.4 Dockerfile:

FROM buildpack-deps:jessie

RUN groupadd --gid 1000 node \
  && useradd --uid 1000 --gid node --shell /bin/bash --create-home node

# gpg keys listed at https://github.com/nodejs/node
RUN set -ex \
  && for key in \
    9554F04D7259F04124DE6B476D5A82AC7E37093B \
    94AE36675C464D64BAFA68DD7434390BDBE9B9C5 \
    0034A06D9D9B0064CE8ADF6BF1747F4AD2306D93 \
    FD3A5288F042B6850C66B31F09FE44734EB7990E \
    71DCFD284A79C3B38668286BC97EC7A07EDE3FC1 \
    DD8F2338BAE7501E3DD5AC78C273792F7D83545D \
    B9AE9905FFD7803F25714661B63B535A4C206CA9 \
    C4F0DFFF4E8C1A8236409D08E73BC641CC11F4C8 \
  ; do \
    gpg --keyserver ha.pool.sks-keyservers.net --recv-keys "$key"; \
  done

ENV NPM_CONFIG_LOGLEVEL info
ENV NODE_VERSION 7.4.0

RUN curl -SLO "https://nodejs.org/dist/v$NODE_VERSION/node-v$NODE_VERSION-linux-x64.tar.xz" \
  && curl -SLO "https://nodejs.org/dist/v$NODE_VERSION/SHASUMS256.txt.asc" \
  && gpg --batch --decrypt --output SHASUMS256.txt SHASUMS256.txt.asc \
  && grep "node-v$NODE_VERSION-linux-x64.tar.xz\$" SHASUMS256.txt | sha256sum -c - \
  && tar -xJf "node-v$NODE_VERSION-linux-x64.tar.xz" -C /usr/local --strip-components=1 \
  && rm "node-v$NODE_VERSION-linux-x64.tar.xz" SHASUMS256.txt.asc SHASUMS256.txt \
  && ln -s /usr/local/bin/node /usr/local/bin/nodejs

CMD [ "node" ]

Both Dockerfiles use the same base image buildpack-deps:jessie. This is excellent because you can combine them and install Python to get awscli.

Remove the associated files before committing the Docker image to install by using apt. You can install everything and remove those files afterward, but do not run apt-get update more than once. Any custom repositories are added beforehand.

The Ruby image comes with git pre-installed so there is no reason to reinstall it. Next, add sudo, python2.7, and postgresql-9.5 to the list installation list. Then, install yarn with npm.

FROM buildpack-deps:jessie

RUN groupadd --gid 1000 node \
  && useradd --uid 1000 --gid node --shell /bin/bash --create-home node

# gpg keys listed at https://github.com/nodejs/node
RUN set -ex \
  && for key in \
    9554F04D7259F04124DE6B476D5A82AC7E37093B \
    94AE36675C464D64BAFA68DD7434390BDBE9B9C5 \
    0034A06D9D9B0064CE8ADF6BF1747F4AD2306D93 \
    FD3A5288F042B6850C66B31F09FE44734EB7990E \
    71DCFD284A79C3B38668286BC97EC7A07EDE3FC1 \
    DD8F2338BAE7501E3DD5AC78C273792F7D83545D \
    B9AE9905FFD7803F25714661B63B535A4C206CA9 \
    C4F0DFFF4E8C1A8236409D08E73BC641CC11F4C8 \
  ; do \
    gpg --keyserver ha.pool.sks-keyservers.net --recv-keys "$key"; \
  done

ENV NPM_CONFIG_LOGLEVEL info
ENV NODE_VERSION 7.4.0
ENV YARN_VERSION 0.18.1

RUN curl -SLO "https://nodejs.org/dist/v$NODE_VERSION/node-v$NODE_VERSION-linux-x64.tar.xz" \
  && curl -SLO "https://nodejs.org/dist/v$NODE_VERSION/SHASUMS256.txt.asc" \
  && gpg --batch --decrypt --output SHASUMS256.txt SHASUMS256.txt.asc \
  && grep "node-v$NODE_VERSION-linux-x64.tar.xz\$" SHASUMS256.txt | sha256sum -c - \
  && tar -xJf "node-v$NODE_VERSION-linux-x64.tar.xz" -C /usr/local --strip-components=1 \
  && rm "node-v$NODE_VERSION-linux-x64.tar.xz" SHASUMS256.txt.asc SHASUMS256.txt \
  && ln -s /usr/local/bin/node /usr/local/bin/nodejs

# Postgres 9.5
RUN echo "deb http://apt.postgresql.org/pub/repos/apt/ jessie-pgdg main" >> /etc/apt/sources.list \
      && wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add - \
      && apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 58118E89F3A912897C070ADBF76221572C52609D 514A2AD631A57A16DD0047EC749D6EEC0353B12C

# skip installing gem documentation
RUN mkdir -p /usr/local/etc \
	&& { \
		echo 'install: --no-document'; \
		echo 'update: --no-document'; \
	} >> /usr/local/etc/gemrc

ENV RUBY_MAJOR 2.1
ENV RUBY_VERSION 2.1.10
ENV RUBY_DOWNLOAD_SHA256 5be9f8d5d29d252cd7f969ab7550e31bbb001feb4a83532301c0dd3b5006e148
ENV RUBYGEMS_VERSION 2.6.10

# some of ruby's build scripts are written in ruby
#   we purge system ruby later to make sure our final image uses what we just built
RUN set -ex \
	\
	&& buildDeps=' \
		bison \
		libgdbm-dev \
		ruby \
	' \
	&& apt-get update \
	&& apt-get install -y --no-install-recommends $buildDeps python2.7 sudo postgresql-9.5 \
	&& rm -rf /var/lib/apt/lists/* \
	\
	&& wget -O ruby.tar.xz "https://cache.ruby-lang.org/pub/ruby/${RUBY_MAJOR%-rc}/ruby-$RUBY_VERSION.tar.xz" \
	&& echo "$RUBY_DOWNLOAD_SHA256 *ruby.tar.xz" | sha256sum -c - \
	\
	&& mkdir -p /usr/src/ruby \
	&& tar -xJf ruby.tar.xz -C /usr/src/ruby --strip-components=1 \
	&& rm ruby.tar.xz \
	\
	&& cd /usr/src/ruby \
	\
# hack in "ENABLE_PATH_CHECK" disabling to suppress:
#   warning: Insecure world writable dir
	&& { \
		echo '#define ENABLE_PATH_CHECK 0'; \
		echo; \
		cat file.c; \
	} > file.c.new \
	&& mv file.c.new file.c \
	\
	&& autoconf \
	&& ./configure --disable-install-doc --enable-shared \
	&& make -j"$(nproc)" \
	&& make install \
	\
	&& apt-get purge -y --auto-remove $buildDeps \
	&& cd / \
	&& rm -r /usr/src/ruby \
	\
	&& gem update --system "$RUBYGEMS_VERSION"

ENV BUNDLER_VERSION 1.14.3

RUN gem install bundler --version "$BUNDLER_VERSION"

RUN npm install -g yarn@0.18.1
ENV PATH "$PATH:/root/.yarn/bin/:/usr/local/bin"

# install things globally, for great justice
# and don't create ".bundle" in all our apps
ENV GEM_HOME /usr/local/bundle
ENV BUNDLE_PATH="$GEM_HOME" \
	BUNDLE_BIN="$GEM_HOME/bin" \
	BUNDLE_SILENCE_ROOT_WARNING=1 \
	BUNDLE_APP_CONFIG="$GEM_HOME"
ENV PATH $BUNDLE_BIN:$PATH
RUN mkdir -p "$GEM_HOME" "$BUNDLE_BIN" \
	&& chmod 777 "$GEM_HOME" "$BUNDLE_BIN"

CMD [ "irb" ]

To build it, run the following command:

docker build -t ruby-node:0.1 .

When it completes, it should display the following:

Removing intermediate container e75339607356
Successfully built 52b773cf50e2

After it finishes compiling, take the SHA from the Docker output and run it as follows:

docker run -it 52b773cf50e2 /bin/bash
root@6cd398c7b61d:/# exit

Then, commit that hostname replacing ruby-node with your Docker Hub username as follows:

docker commit 6cd398c7b61d username/ruby-node:0.1
docker push username/ruby-node:0.1

To use the custom image, reference ruby-node/bar:0.1 in your .circleci/config.yml image key and your primary container will run it. It is worth it to commit your Dockerfile using a gist and link to it from Docker Hub to avoid losing your configuration.