Use the CircleCI environment CLI to split tests
CircleCI supports automatic test allocation across parallel compute environments. When the parallelism key in your CircleCI configuration is set to a value greater than 1, CircleCI spins up the specified number of identical execution environments in which to run your job.
Test splitting requires the CircleCI environment CLI together with parallelism. The environment CLI commands circleci tests glob and circleci tests split are used to define your test suite and allocate tests across multiple environments. The environment CLI is automatically injected into your job at run-time, so there is no further setup required to use the circleci tests commands.
The circleci tests commands are part of the environment CLI as they require information that only exists within a CircleCI container. circleci tests commands are not available in the CircleCI CLI that you install locally.
|
Self-Hosted Runners can invoke circleci-agent directly instead of using the CLI to split tests. The task-agent already exists on the $PATH, which removes an additional dependency when splitting tests.
|
On CircleCI Server, use circleci tests glob and circleci tests split. circleci tests run is not supported on CircleCI Server, even though it is part of the same command family. See Using tests split below.
|
Prerequisites
-
A CircleCI Server account connected to your code.
-
A project in CircleCI where you run tests. See the Create a Project page to get set up, and Upload Test Results page for information about tests in your CI/CD pipeline.
Using tests split
Optimize your tests using multiple parallel-running environments. First, glob your test files, then split test suite, then run your tests.
1. Glob test files
Use circleci tests glob to define your test suite. To glob test files, pass one or more patterns to the glob command:
circleci tests glob "tests/unit/*.java" "tests/functional/*.java"The environment CLI supports globbing test files using the following patterns:
-
*matches any sequence of characters (excluding path separators). -
**matches any sequence of characters (including path separators). -
?matches any single character (excluding path separators). -
[abc]matches any character (excluding path separators) against characters in brackets. -
{foo,bar,…}matches a sequence of characters, if any of the alternatives in braces matches.
Ensure that the glob string has quotes. To check the results of pattern-matching, use the echo command.
# ~/.circleci/config.yml
version: 2.1
jobs:
test:
docker:
- image: cimg/node:20.3.0
parallelism: 4
steps:
- run:
command: |
echo $(circleci tests glob "foo/**/*" "bar/**/*")
circleci tests glob "foo/**/*" "bar/**/*" | xargs -n 1 echo2. Split tests
To split your tests, pass in a list of tests to the circleci tests split command.
The following test splitting options are available:
-
Alphabetically by name (default if none specified).
-
Split using timing data
--split-by=timings. We recommend this option as it results in the most even split across your parallel execution environments. -
Split using file size
--split-by=filesize.
a. Split by name (default)
By default, if you do not specify a method using the --split-by flag, circleci tests split expects a list of file names or class names and splits tests alphabetically by test name. You can provide this list in the following ways:
-
Pipe a glob of test files, as demonstrated in the above section.
circleci tests glob "test/**/*.java" | circleci tests split-
Create a text file with test filenames.
circleci tests split test_filenames.txt-
Provide a path to the test files.
circleci tests split < /path/to/items/to/splitb. Split by timing data
The best way to optimize your test suite across a set of parallel executors is to split your tests using timing data. This will ensure the tests are split in the most even way, leading to a shorter test time.
To split by test timing, use the --split-by flag with the timings split type.
circleci tests glob "**/*.go" | circleci tests split --split-by=timings
If you do not use store_test_results, there will be no timing data available to split your tests.
|
On each successful run of a test suite, CircleCI saves timing data from the directory specified by the path in the store_test_results step. This timing data consists of how long each test took to complete per file name or class name.
The available timing data will then be analyzed and your tests will be split across your parallel-running containers as evenly as possible.
If no timing data is found, you will receive a message: Error auto-detecting timing type, falling back to weighting by name.. The tests will then be split alphabetically by test name.
|
Set the timing type
The environment CLI attempts to autodetect the granularity of the test split (for example, whether to split by filename, or down to class name) based on the input. You may need to choose a different timing type depending on how your test coverage output is formatted, using the --timings-type option. Valid timing types are:
-
filename -
classname -
testname -
autodetect
cat my_java_test_classnames | circleci tests split --split-by=timings --timings-type=classnameSet the default value for missing timing data
For partially found test results, any tests with missing data are assigned a random small value. You can override this default value with the --time-default flag:
circleci tests glob "**/*.rb" | circleci tests split --split-by=timings --time-default=10sDownload timing data
If you need to manually store and retrieve timing data, add the store_artifacts step to your job.
3. Run split tests
Globbing and splitting tests does not actually run your tests. To combine test grouping with test execution, consider saving the grouped tests to a file, then passing this file to your test runner.
circleci tests glob "test/**/*.rb" | circleci tests split > /tmp/tests-to-run
bundle exec rspec $(cat /tmp/tests-to-run)The contents of the file /tmp/tests-to-run will be different in each container, based on $CIRCLE_NODE_INDEX and $CIRCLE_NODE_TOTAL.