As a Success Engineer, I often see a lot of confusion around using Docker on CircleCI. Before the release of CircleCI 2.0, it was possible to build and test applications without ever having to think about Docker: all our builds ran in LXC containers, configured by our inference engine to include any necessary tools and dependencies in an oftentimes fairly automatic way.
With the release of CircleCI 2.0, however, we’ve placed Docker front-and-center in our build process: any Docker image, public or private (we now support private images hosted on Amazon’s EC2 Container Registry, too!), can be the primary container for a build. This adds power and customization to the build process, especially when combined with Workflows, which allows for sets of jobs (build, test, deploy, etc.) to be strung together, sometimes running in parallel, each with its own base Docker image. To aid the transition from CircleCI 1.0 to 2.0, we’ve even provided a wide range of convenience images to meet common language and dependency needs, tweaked to maximize compatibility with the CircleCI platform.
However, as they say, with great power comes great responsibility. Using Docker can be intimidating. For some developers, it may seem to have a high learning curve. Others worry about maintenance or upkeep for any images they’ve created. And then there are privacy concerns—after all, many of the images on the main Docker registry, Docker Hub, are completely public. But Docker can be a valuable ally as you build and test projects on CircleCI 2.0. Below, we’ve outlined a couple common issues folks encounter using CircleCI 2.0, and how Docker can help solve them.
Problem 1: Builds take too long
There are any number of reasons a build might take a long time to run on CircleCI. But one common factor is installing dependencies. As I’m helping customers migrate projects to CircleCI 2.0, I often start by adding a couple of missing packages here and there. Soon, though, this can turn into a whole series of
run steps, and, even with caching enabled, can add a couple of minutes to a large project’s build time. One way to alleviate this situation is by offloading some of these dependency installations into your Docker image.
Your Docker image might feel like something separate from your CircleCI build, but it can be helpful to think about it as the first step of your build process. If you’re installing the same couple of packages every single time you run your build, there’s no reason you can’t install them into the Docker image instead. That way, they’ll be pre-installed each time you start a build. As an example, let’s say we’re using a CircleCI convenience image. From there, it’s not so hard to create a Dockerfile and add a couple of simple steps to it to install our dependencies. Your Dockerfile might look something like this:
FROM circleci/ruby:2.4.1-node-browsers USER root # pulling in commands from 'install os packages' RUN apt-get update RUN apt-get install -y qt5-default libqt5webkit5-dev gstreamer1.0-plugins-base gstreamer1.0-tools gstreamer1.0-x postgresql-client USER circleci
This tells Docker to start with our
circleci/ruby:2.4.1-node-browsers convenience image, switch to
root to install a couple of dependencies, then switch back to
circleci (the default user for all of our convenience images).
Assuming you’ve created a Docker ID and installed Docker on your computer, then all you need to do is build your image locally and push it to the Docker Hub:
docker build -t my-docker-id/my-image:my-tag /path/to/Dockerfile
docker push my-docker-id/my-image:my-tag
Now you can use your new image for your build, and delete those
apt-get install commands from your
config.yml file, shaving valuable seconds or minutes off your build time as well as keeping your
config.yml code lean and dry. See our walkthrough of this process for more, or check out Docker’s guide.
Problem 2: I need a specific legacy version of [X]
With our convenience images, we try to cover the most popular versions of most languages and software development environments. But sometimes you might need a particular version of, say, Node.js for your build—what if we don’t offer it?
This is a fairly common problem we run into working with customers on CircleCI 2.0. Without access to their source code, it’s hard to know whether using a different version of a particular dependency will break their build or not, so I usually err on the side of caution and try to replicate their 1.0 execution environment as closely as possible. In this case, all that’s needed are a couple of lines of text in a Dockerfile to take our convenience Node.js image and downgrade to a different version:
FROM circleci/node:6.11 USER root # delete node 6.11 files RUN rm -f /usr/local/bin/node /usr/local/bin/nodejs /usr/local/bin/npm /usr/local/share/man/man1/node.1 /usr/local/share/systemtap/tapset/node.stp RUN rm -rf /usr/local/lib/node_modules /usr/local/include/node /usr/local/share/doc/node # download node 6.8.1 from here: https://nodejs.org/en/download/releases RUN wget https://nodejs.org/download/release/v6.8.1/node-v6.8.1.tar.gz RUN tar xvf node-v6.8.1.tar.gz WORKDIR node-v6.8.1 # install node 6.8.1 as described here: https://github.com/nodejs/node/blob/master/BUILDING.md#building-nodejs-on-supported-platforms RUN ./configure RUN make -j4 RUN make doc RUN make install USER circleci
Here, we start with Node version 6.11, delete all relevant Node 6.11 files and directories, and download/compile/install Node version 6.8.1. In this case, there are a couple of Node version management tools that might make this task easier (n or nvm are popular choices), or we could try using Ubuntu’s
apt-get package management system, but it’s always felt safer to me to start fresh and compile from source.
A simple Google search can help identify the upgrade/downgrade process for whatever language or tool you’re using, and at the end of the day, you still get the benefit of our convenience images’ optimizations for running on CircleCI. From there, the process is the same: build your image, push it to the Docker Hub, and put it in your
Problem 3: But wait, I don’t want to build Docker images on my local machine
That’s actually fine, because with CircleCI 2.0, you don’t have to. Instead, you can use our
machine executor to spin up a full VM for your build and create Docker images within CircleCI to your heart’s content. Or use the
docker executor in tandem with the
setup_remote_docker key—we actually published an entire blog post about this approach back in June.
You could even set up an entire repository just to build Docker images—that’s how we do it. And with the upcoming release of scheduled jobs, you’ll be able to build your images using our resources, regularly, with no maintenance on your end.
Problem 4: I don’t want my Docker images to be public
Your Docker images can contain lots of private or sensitive data, and you might not want that to be accessible to the public.
Luckily, Docker Hub supports private Docker images—it’s as easy as changing the default visibility of your images in your Docker Hub account settings. You can even take things a step further and use an entirely private Docker registry instead of Docker Hub.
Ready for more? Read the 2nd blog post in this series on how you can take your newfound Docker skills to the next level with Workflows. Or see this great Discuss post from Senior Customer Success Engineer Ryan O’Hara for a more in-depth look at building Docker images.