Kegoneko-Balance

Tl;DR: Instead of naively splitting tests among build containers, CircleCI now divides tests intelligently based on execution time.

Status quo on CircleCI

For years, CircleCI has enabled fast execution of large test suites by splitting tests evenly across many build containers (LXC containers under the hood–the same kind of lightweight containerization as Docker uses). While we have always provided the option to set up parallel actions by manually specifying what should happen on each container, the simplest way to use CircleCI’s parallelism is to allow our automatically inferred test steps to run your tests in parallel. Tests are distributed amongst containers so that an equal number of tests runs on each, and all of this happens without any configuration for standard project setups using popular frameworks and languages.

The new hotness

Now our automatic parallelism goes one amazing step further, automatically dividing tests across containers weighted by past execution time instead of just number of tests, allowing your test suite to run even faster. For example, if you have 90 browser tests that take 10 seconds each and 100 unit tests that take 1 second each, all run by RSpec, then with 10X parallelism CircleCI will spread the browser tests across 9 containers and run all the unit tests on 1 (or otherwise evenly distribute them) so that each container spends an equal amount of time running tests. All of this happens automatically, without any extra setup required, and your tests will be continually rebalanced to run as fast as possible as your test suite grows.

How to turn it on

Automatic test balancing is currently supported for the following test runners:

  • RSpec

  • minitest

  • nose

  • Cucumber

As long as your tests run with CircleCI’s inferred commands, no additional configuration is needed, though RSpec and minitest do require that you simply include a special test formatter gem in your project’s dependencies. See this doc for details.

How it works

Under the hood, the new automatic test balancing uses the same JUnit-formatted test output that enables our detailed test failure reporting. CircleCI parses and stores the timing data for each test. That information is then used to perform a time-weighted distribution of tests across build containers, which is then applied by passing command line flags to each test runner.

Next steps

We will be working on adding support for more test runners soon–please drop us a line at sayhi@circleci.com if you’d like to request support for a particular runner! We will also work on exposing an interface to the previous test timing data so that the same auto-balancing can be customized to work with less common test suites. Happy testing!