Start Building for Free
CircleCI.comAcademyBlogCommunitySupport

Use the CircleCI CLI to split tests

2 weeks ago3 min read
Cloud
Server v4.x
Server v3.x
On This Page

CircleCI supports automatic test allocation across parallel compute environments. When the parallelism key in your CircleCI configuration is set to a value greater than 1, CircleCI spins up identical execution environments in which your job is run.

Test splitting requires the CircleCI CLI together with parallelism. The CLI commands circleci tests glob and circleci tests split are used to define your test suite and allocate tests across multiple environments. The CLI is automatically injected into your job at run-time, so there is no further setup required to use the circleci tests commands.

1. Glob test files

Use circleci tests glob to define your test suite. To glob test files, pass one or more patterns to the glob command:

circleci tests glob "tests/unit/*.java" "tests/functional/*.java"

The CLI supports globbing test files using the following patterns:

  • * matches any sequence of characters (excluding path separators)

  • ** matches any sequence of characters (including path separators)

  • ? matches any single character (excluding path separators)

  • [abc] matches any character (excluding path separators) against characters in brackets

  • {foo,bar,…​} matches a sequence of characters, if any of the alternatives in braces matches

Ensure that the glob string has quotes. To check the results of pattern-matching, use the echo command.

# ~/.circleci/config.yml
version: 2.1
jobs:
  test:
    docker:
      - image: cimg/<language>:<version TAG>
        auth:
          username: mydockerhub-user
          password: $DOCKERHUB_PASSWORD  # context / project UI env-var reference
    parallelism: 4
    steps:
      - run:
          command: |
            echo $(circleci tests glob "foo/**/*" "bar/**/*")
            circleci tests glob "foo/**/*" "bar/**/*" | xargs -n 1 echo

2. Split tests

To split tests, pass in a list of tests to the circleci tests split command.

The following test splitting options are available:

  • Alphabetically by name (by default, if none specified)

  • --split-by=timings (recommended)

  • --split-by=filesize

a. Split by name (default)

By default, if you do not specify a method using the --split-by flag, circleci tests split expects a list of filenames or classnames and splits tests alphabetically by test name. There are a few ways to provide this list:

  • Pipe a glob of test files, as demonstrated in the above section.

circleci tests glob "test/**/*.java" | circleci tests split
  • Create a text file with test filenames.

circleci tests split test_filenames.txt
  • Provide a path to the test files.

circleci tests split < /path/to/items/to/split

b. Split by timing data

The best way to optimize your test suite across a set of parallel executors is to split your tests using timing data. This will ensure the tests are split in the most even way, leading to a shorter test time.

To split by test timing, use the --split-by flag with the timings split type.

circleci tests glob "**/*.go" | circleci tests split --split-by=timings

On each successful run of a test suite, CircleCI saves timing data from the directory specified by the path in the store_test_results step. This timing data consists of how long each test took to complete per filename or classname.

The available timing data will then be analyzed and your tests will be split across your parallel-running containers as evenly as possible.

JUnit XML reports

CircleCI requires test results to be uploaded as JUnit XML reports. The following formatting allows CircleCI to parse timing data from test results and use the data for test splitting:

  • The file attribute, either on the <testsuite> or <testcase> tag

  • The time attribute, on the <testcase> tag

The following example is a snippet from an XML file with a format that CircleCI can parse:

<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="Mocha Tests" tests="3" failures="1">
  <testsuite tests="3">
    <testcase classname="foo1" name="ASuccessfulTest" time="10" file="src/__tests__/App.test.js" />
    <testcase classname="foo2" name="AnotherSuccessfulTest" time="5" file="src/__tests__/App.test.js" />
    <testcase classname="foo3" name="AFailingTest" time="1.1050" file="src/__tests__/App.test.js">
        <failure type="NotEnoughFoo"> details about failure </failure>
    </testcase>
  </testsuite>
</testsuites>

Set the timing type

The CLI attempts to autodetect the granularity of the test split (for example, whether to split by filename, or down to classname) based on the input to the split command. You may need to choose a different timing type depending on how your test coverage output is formatted, using the --timings-type option. Valid timing types are:

  • filename

  • classname

  • testname

  • autodetect

cat my_java_test_classnames | circleci tests split --split-by=timings --timings-type=classname

Set the default value for missing timing data

For partially found test results, any tests with missing data are assigned a random small value. You can override this default value with the --time-default flag:

circleci tests glob "**/*.rb" | circleci tests split --split-by=timings --time-default=10s

Download timing data

If you need to manually store and retrieve timing data, add the store_artifacts step to your job.

c. Split by filesize

When provided with filepaths, the CLI can also split by filesize. Use the --split-by flag with the filesize split type:

circleci tests glob "**/*.go" | circleci tests split --split-by=filesize

3. Run split tests

Globbing and splitting tests does not actually run your tests. To combine test grouping with test execution, consider saving the grouped tests to a file, then passing this file to your test runner.

circleci tests glob "test/**/*.rb" | circleci tests split > /tmp/tests-to-run
bundle exec rspec $(cat /tmp/tests-to-run)

The contents of the file /tmp/tests-to-run will be different in each container, based on $CIRCLE_NODE_INDEX and $CIRCLE_NODE_TOTAL.


Help make this document better

This guide, as well as the rest of our docs, are open source and available on GitHub. We welcome your contributions.

Need support?

Our support engineers are available to help with service issues, billing, or account related questions, and can help troubleshoot build configurations. Contact our support engineers by opening a ticket.

You can also visit our support site to find support articles, community forums, and training resources.