Use the CircleCI CLI to split tests
CircleCI supports automatic test allocation across parallel compute environments. When the parallelism
key in your CircleCI configuration is set to a value greater than 1
, CircleCI spins up identical execution environments in which your job is run.
Test splitting requires the CircleCI CLI together with parallelism. The CLI commands circleci tests glob
and circleci tests split
are used to define your test suite and allocate tests across multiple environments. The CLI is automatically injected into your job at run-time, so there is no further setup required to use the circleci tests
commands.
The circleci tests commands cannot be run locally via the CLI as they require information that only exists within a CircleCI container. |
Self-hosted runners can invoke circleci-agent directly instead of using the CLI to split tests. This is because the task-agent already exists on the $PATH , removing an additional dependency when splitting tests. |
1. Glob test files
Use circleci tests glob
to define your test suite. To glob test files, pass one or more patterns to the glob
command:
circleci tests glob "tests/unit/*.java" "tests/functional/*.java"
The CLI supports globbing test files using the following patterns:
-
*
matches any sequence of characters (excluding path separators) -
**
matches any sequence of characters (including path separators) -
?
matches any single character (excluding path separators) -
[abc]
matches any character (excluding path separators) against characters in brackets -
{foo,bar,…}
matches a sequence of characters, if any of the alternatives in braces matches
Ensure that the glob string has quotes. To check the results of pattern-matching, use the echo
command.
# ~/.circleci/config.yml
version: 2.1
jobs:
test:
docker:
- image: cimg/<language>:<version TAG>
auth:
username: mydockerhub-user
password: $DOCKERHUB_PASSWORD # context / project UI env-var reference
parallelism: 4
steps:
- run:
command: |
echo $(circleci tests glob "foo/**/*" "bar/**/*")
circleci tests glob "foo/**/*" "bar/**/*" | xargs -n 1 echo
2. Split tests
To split tests, pass in a list of tests to the circleci tests split
command.
The following test splitting options are available:
-
Alphabetically by name (by default, if none specified)
-
--split-by=timings
(recommended) -
--split-by=filesize
a. Split by name (default)
By default, if you do not specify a method using the --split-by
flag, circleci tests split
expects a list of filenames or classnames and splits tests alphabetically by test name. There are a few ways to provide this list:
-
Pipe a glob of test files, as demonstrated in the above section.
circleci tests glob "test/**/*.java" | circleci tests split
-
Create a text file with test filenames.
circleci tests split test_filenames.txt
-
Provide a path to the test files.
circleci tests split < /path/to/items/to/split
b. Split by timing data
The best way to optimize your test suite across a set of parallel executors is to split your tests using timing data. This will ensure the tests are split in the most even way, leading to a shorter test time.
To split by test timing, use the --split-by
flag with the timings
split type.
circleci tests glob "**/*.go" | circleci tests split --split-by=timings
If you do not use store_test_results , there will be no timing data available to split your tests. |
On each successful run of a test suite, CircleCI saves timing data from the directory specified by the path in the store_test_results
step. This timing data consists of how long each test took to complete per filename or classname.
The available timing data will then be analyzed and your tests will be split across your parallel-running containers as evenly as possible.
If no timing data is found, you will receive a message: Error auto-detecting timing type, falling back to weighting by name. . The tests will then be split alphabetically by test name. |
JUnit XML reports
CircleCI requires test results to be uploaded as JUnit XML reports. The following formatting allows CircleCI to parse timing data from test results and use the data for test splitting:
-
The
file
attribute, either on the<testsuite>
or<testcase>
tag -
The
time
attribute, on the<testcase>
tag
The following example is a snippet from an XML file with a format that CircleCI can parse:
<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="Mocha Tests" tests="3" failures="1">
<testsuite tests="3">
<testcase classname="foo1" name="ASuccessfulTest" time="10" file="src/__tests__/App.test.js" />
<testcase classname="foo2" name="AnotherSuccessfulTest" time="5" file="src/__tests__/App.test.js" />
<testcase classname="foo3" name="AFailingTest" time="1.1050" file="src/__tests__/App.test.js">
<failure type="NotEnoughFoo"> details about failure </failure>
</testcase>
</testsuite>
</testsuites>
Set the timing type
The CLI attempts to autodetect the granularity of the test split (for example, whether to split by filename, or down to classname) based on the input to the split
command. You may need to choose a different timing type depending on how your test coverage output is formatted, using the --timings-type
option. Valid timing types are:
-
filename
-
classname
-
testname
-
autodetect
cat my_java_test_classnames | circleci tests split --split-by=timings --timings-type=classname
Set the default value for missing timing data
For partially found test results, any tests with missing data are assigned a random small value. You can override this default value with the --time-default
flag:
circleci tests glob "**/*.rb" | circleci tests split --split-by=timings --time-default=10s
Download timing data
If you need to manually store and retrieve timing data, add the store_artifacts
step to your job.
c. Split by filesize
When provided with filepaths, the CLI can also split by filesize. Use the --split-by
flag with the filesize
split type:
circleci tests glob "**/*.go" | circleci tests split --split-by=filesize
3. Run split tests
Globbing and splitting tests does not actually run your tests. To combine test grouping with test execution, consider saving the grouped tests to a file, then passing this file to your test runner.
circleci tests glob "test/**/*.rb" | circleci tests split > /tmp/tests-to-run
bundle exec rspec $(cat /tmp/tests-to-run)
The contents of the file /tmp/tests-to-run
will be different in each container, based on $CIRCLE_NODE_INDEX
and $CIRCLE_NODE_TOTAL
.
Help make this document better
This guide, as well as the rest of our docs, are open source and available on GitHub. We welcome your contributions.
- Suggest an edit to this page (please read the contributing guide first).
- To report a problem in the documentation, or to submit feedback and comments, please open an issue on GitHub.
- CircleCI is always seeking ways to improve your experience with our platform. If you would like to share feedback, please join our research community.
Need support?
Our support engineers are available to help with service issues, billing, or account related questions, and can help troubleshoot build configurations. Contact our support engineers by opening a ticket.
You can also visit our support site to find support articles, community forums, and training resources.
CircleCI Documentation by CircleCI is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.