Use the CircleCI CLI to split tests

Cloud

Server v4.x

Server v3.x

Test splitting requires the CircleCI CLI together with parallelism. The CLI commands circleci tests glob and circleci tests split are used to define your test suite and allocate tests across multiple environments. The CLI is automatically injected into your job at run-time, so there is no further setup required to use the circleci tests commands.

The circleci tests commands cannot be run locally via the CLI as they require information that only exists within a CircleCI container.

Self-hosted runners can invoke circleci-agent directly instead of using the CLI to split tests. This is because the task-agent already exists on the $PATH, removing an additional dependency when splitting tests.

1. Glob test files

Use circleci tests glob to define your test suite. To glob test files, pass one or more patterns to the glob command:

circleci tests glob "tests/unit/*.java" "tests/functional/*.java"

The CLI supports globbing test files using the following patterns:

* matches any sequence of characters (excluding path separators)
** matches any sequence of characters (including path separators)
? matches any single character (excluding path separators)
[abc] matches any character (excluding path separators) against characters in brackets
{foo,bar,…} matches a sequence of characters, if any of the alternatives in braces matches

Ensure that the glob string has quotes. To check the results of pattern-matching, use the echo command.

# ~/.circleci/config.yml
version: 2.1
jobs:
  test:
    docker:
      - image: cimg/<language>:<version TAG>
        auth:
          username: mydockerhub-user
          password: $DOCKERHUB_PASSWORD  # context / project UI env-var reference
    parallelism: 4
    steps:
      - run:
          command: |
            echo $(circleci tests glob "foo/**/*" "bar/**/*")
            circleci tests glob "foo/**/*" "bar/**/*" | xargs -n 1 echo

2. Split tests

To split tests, pass in a list of tests to the circleci tests split command.

The following test splitting options are available:

Alphabetically by name (by default, if none specified)
--split-by=timings (recommended)
--split-by=filesize

a. Split by name (default)

By default, if you do not specify a method using the --split-by flag, circleci tests split expects a list of filenames or classnames and splits tests alphabetically by test name. There are a few ways to provide this list:

Pipe a glob of test files, as demonstrated in the above section.

circleci tests glob "test/**/*.java" | circleci tests split

Create a text file with test filenames.

circleci tests split test_filenames.txt

Provide a path to the test files.

circleci tests split < /path/to/items/to/split

b. Split by timing data

The best way to optimize your test suite across a set of parallel executors is to split your tests using timing data. This will ensure the tests are split in the most even way, leading to a shorter test time.

To split by test timing, use the --split-by flag with the timings split type.

circleci tests glob "**/*.go" | circleci tests split --split-by=timings

If you do not use store_test_results, there will be no timing data available to split your tests.

On each successful run of a test suite, CircleCI saves timing data from the directory specified by the path in the store_test_results step. This timing data consists of how long each test took to complete per filename or classname.

The available timing data will then be analyzed and your tests will be split across your parallel-running containers as evenly as possible.

If no timing data is found, you will receive a message: Error auto-detecting timing type, falling back to weighting by name.. The tests will then be split alphabetically by test name.

JUnit XML reports

CircleCI requires test results to be uploaded as JUnit XML reports. The following formatting allows CircleCI to parse timing data from test results and use the data for test splitting:

The file attribute, either on the <testsuite> or <testcase> tag
The time attribute, on the <testcase> tag

The following example is a snippet from an XML file with a format that CircleCI can parse:

<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="Mocha Tests" tests="3" failures="1">
  <testsuite tests="3">
    <testcase classname="foo1" name="ASuccessfulTest" time="10" file="src/__tests__/App.test.js" />
    <testcase classname="foo2" name="AnotherSuccessfulTest" time="5" file="src/__tests__/App.test.js" />
    <testcase classname="foo3" name="AFailingTest" time="1.1050" file="src/__tests__/App.test.js">
        <failure type="NotEnoughFoo"> details about failure </failure>
    </testcase>
  </testsuite>
</testsuites>

Set the timing type

The CLI attempts to autodetect the granularity of the test split (for example, whether to split by filename, or down to classname) based on the input to the split command. You may need to choose a different timing type depending on how your test coverage output is formatted, using the --timings-type option. Valid timing types are:

filename
classname
testname
autodetect

cat my_java_test_classnames | circleci tests split --split-by=timings --timings-type=classname

Set the default value for missing timing data

For partially found test results, any tests with missing data are assigned a random small value. You can override this default value with the --time-default flag:

circleci tests glob "**/*.rb" | circleci tests split --split-by=timings --time-default=10s

Download timing data

If you need to manually store and retrieve timing data, add the store_artifacts step to your job.

c. Split by filesize

When provided with filepaths, the CLI can also split by filesize. Use the --split-by flag with the filesize split type:

circleci tests glob "**/*.go" | circleci tests split --split-by=filesize

3. Run split tests

Globbing and splitting tests does not actually run your tests. To combine test grouping with test execution, consider saving the grouped tests to a file, then passing this file to your test runner.

circleci tests glob "test/**/*.rb" | circleci tests split > /tmp/tests-to-run
bundle exec rspec $(cat /tmp/tests-to-run)

The contents of the file /tmp/tests-to-run will be different in each container, based on $CIRCLE_NODE_INDEX and $CIRCLE_NODE_TOTAL.