This post was originally published on Intuit Quickbooks’ engineering blog.
Docker on CI is powerful
CircleCI has an amazing set of features in a simple package that allows our developers to move faster than ever before. Coming from our old build system, we wanted to push Circle to the limits such that we could squeeze as much speed out of it as possible.
- Be as small as possible
- Be reliable/reproducible
- Be fault tolerant
From the outset, we wanted to make sure that we could build at scale with the least amount of friction possible. That meant embracing open source, and interweaving it our internal solutions. At each step of our build process, we focused on these things.
Size Matters (for Docker Containers) 📦
CircleCI leverages Docker containers to build and run the repositories being resolved. This is great because it allows us the flexibility to create containers that are unique to our organization, or to use those from the outside world. Leveraging Alpine containers with basic things baked in like Node, Bash, Yarn etc. provided a nice small start. Unfortunately, these didn’t scale well because it meant having to keep these images updated and add to them for each dependency needed.
Some projects require tools like node-sass, which requires gcc to compile. This would mean more images to make and maintain. Eventually, we decided to use CircleCI’s Public Docker Images. Not only do we not have to make/maintain them, but they also include some nifty hacks baked (e.g. supporting Chrome and Firefox headless on CI when you don’t have a display to output to).
Once we settled on our containers, CircleCI does the rest. Each build attempts to check if the current builder box has the image needed. As we scale, projects will overlap with image requirements. That means that each box will only have to download the image once, and then fetch it when needed. A build only spends time downloading an image when it doesn’t exist. In the worst case, we spend 22s downloading images per build. In the best case, we spend 0s downloading images per build. On a scale of hundreds to thousands of builds an hour using the same images, we avoid spending time downloading and setting up our environment over and over again.
Gotta Cache ’em All ♻️
Many of our builds are NPM modules. This means huge node_modules folders. We are talking 500MBs large. Each build needs to redownload all of those and construct the file hierarchy. Thankfully, updating dependencies is not something that happens often. As such, we can avoid this all together.
CircleCI allows us to specify caching of folders and files.Utilizing this is important as it prevents the need to do repetitive tasks. At the top of each .circleci/config.yml file, we provide the following:
And we restore using:
This allows us to always have a cache key. This is important because it means we will never have to download the entire node_modules directory at any point in the build cycle after the initial build of the project. CircleCI will attempt to match the key in descending order starting at the top. Thus, we provide most specific to least specific. This is useful when having PRs from forks or branches because it means the branch will always have a base to work off of. At any point, we will have some node_modules already downloaded. The diff for Yarn or NPM will be negligible, if one is needed.
When we save, we use the following:
This provides us with both the node_modules we need, and the Yarn cache for faster resolutions should we need to download new dependencies.
In the worst case, this costs us 2 minutes and 18 seconds to download everything fresh via Yarn. When we have a cache, it only takes us 24 seconds to download our entire directory, and 0 seconds to resolve via Yarn or NPM.
As an added bonus, if your internal registry ever goes down, builds can proceed because you cached your dependencies.
CircleCI turned out to be more performant in general than our prior systems. One of the main areas we saved time on was queueing builds. Our prior system required the provisioning of a builder node. This node could take 1 minute and 30 seconds or longer to provision, and that is before any container was resolved.
With CircleCI, we don’t experience this. Builds are immediate thanks to their use of Docker and other technologies. Again, having this on multiple pull requests and builds over the course of the day saves hours of dev time, if not days.
How much time did we end up saving? On our prior system, our average build time for the repositories we were measuring was 6 minutes and 34 seconds.
CircleCI, with caching and all the goodies above, brought that time down to 3 minutes and 33 seconds.
- By having local Docker images cached and ready, we shaved 1 minute and 30 seconds off each build.
- Caching node_modules decreased resolution time from 2 minutes and 24 seconds to just 24 seconds, saving 2 minutes on each build.
- By sharing Docker images, we removed the need to redownload and set up the environment every time.
Bringing it Full CI/CD 🚀
The ability to save/restore/propagate artifacts between builds is an enormous time save. While we are just scratching the surface, these initial changes have helped to accelerate our development speeds hand over fist. However, there are still more improvements to be made.
Looking to the future, we will start to leverage Docker locally to vet builds before they go to CI. While limited in scope, CircleCI allows you to do this today for building locally using their CLI tool. We found this to be a bit limiting since it was only building, and not testing. One of our engineers ended up making a .circleci/config-local.yml to pass into the CLI tool. This local .yml contained the test flow, but named under “build” so the tool would run it. Using npm-scripts and Husky, the engineer was able to get tests to run in Docker with the same container that would be on CI via a prepush hook. This will allow contributors to know that their changes will work when they get to CI.
In addition, some folks have been asking about shallow clones for larger repositories. While CircleCI doesn’t natively support this out of box, we have hacked together a solution (based off of their checkout script) that allowed us to do shallow clones. For reference, we were able to decrease a 1.5 GB clone that took 7 minutes to just a few megabytes that took 10 seconds. You can see (and give feedback on) the changes on GitHub.