Running Tests in Parallel
If your project has a large number of tests, it will need more time to run them on one machine. To reduce this time, you can run tests in parallel by spreading them across multiple machines. This requires specifying a parallelism level. You can use either the CircleCI CLI to split test files, or use environment variables to configure each parallel machine individually.
- Specifying a Job’s Parallelism Level
- Using the CircleCI CLI to Split Tests
- Using Environment Variables to Split Tests
- Running Split Tests
- See Also
- Other ways to split tests
Specifying a Job’s Parallelism Level
Test suites are conventionally defined at the job level in your .circleci/config.yml
file.
The parallelism
key specifies
how many independent executors will be set up
to run the steps of a job.
To run a job’s steps in parallel,
set the parallelism
key to a value greater than 1.
# ~/.circleci/config.yml
version: 2
jobs:
test:
docker:
- image: circleci/<language>:<version TAG>
parallelism: 4
For more information, see the Configuring CircleCI document.
Using the CircleCI CLI to Split Tests
CircleCI supports automatic test allocation across your containers. The allocation is file-based. It requires the CircleCI CLI, which is automatically injected into your build at run-time.
To install the CLI locally, see the Using the CircleCI Local CLI document.
Globbing Test Files
The CLI supports globbing test files using the following patterns:
*
matches any sequence of characters (excluding path separators)**
matches any sequence of characters (including path separators)?
matches any single character (excluding path separators)[abc]
matches any character (excluding path separators) against characters in brackets{foo,bar,...}
matches a sequence of characters, if any of the alternatives in braces matches
To glob test files,
pass one or more patterns to the circleci tests glob
command.
circleci tests glob "tests/unit/*.java" "tests/functional/*.java"
To check the results of pattern-matching,
use the echo
command.
# ~/.circleci/config.yml
version: 2
jobs:
test:
docker:
- image: circleci/<language>:<version TAG>
parallelism: 4
steps:
- run:
command: |
echo $(circleci tests glob "foo/**/*" "bar/**/*")
circleci tests glob "foo/**/*" "bar/**/*" | xargs -n 1 echo
Splitting Test Files
The CLI supports splitting tests across machines
when running parallel builds.
To do this,
pass a list of filenames to the circleci tests split
command.
circleci tests split test_filenames.txt
The CLI looks up the number of containers, along with the current container index. Then, it uses deterministic splitting algorithms to split the test files across all available containers.
By default,
the number of containers is specified by the parallelism
key.
You can manually set this
by using the --total
flag.
circleci tests split --total=4 test_filenames.txt
Similarly,
the current container index is automatically picked up from environment variables,
but can be manually set
by using the --index
flag.
circleci tests split --index=0 test_filenames.txt
Splitting by Name
By default,
circleci tests split
expects a list of filenames
and splits tests alphabetically by test name.
There are a few ways to provide this list:
Create a text file with test filenames.
circleci tests split test_filenames.txt
Provide a path to the test files.
circleci tests split < /path/to/items/to/split
Or pipe a glob of test files.
circleci tests glob "test/**/*.java" | circleci tests split
Splitting by Filesize
When provided with filepaths,
the CLI can also split by filesize.
To do this,
use the --split-by
flag with the filesize
split type.
circleci tests glob "**/*.go" | circleci tests split --split-by=filesize
Splitting by Timings Data
On each successful run of a test suite,
CircleCI saves timings data to a directory
specified by the path in the store_test_results
step.
If you do not use store_test_results
,
there will be no timing data available for splitting your tests.
To split by test timings,
use the --split-by
flag with the timings
split type.
circleci tests glob "**/*.go" | circleci tests split --split-by=timings
The CLI expects both filenames and classnames
to be present in the timing data
produced by the testing suite.
By default, splitting defaults to filename,
but you can specify classnames
by using the --timings-type
flag.
cat my_java_test_classnames | circleci tests split --split-by=timings --timings-type=classname
If you need to manually store and retrieve timing data,
use the store_artifacts
step.
Using Environment Variables to Split Tests
For full control over parallelism,
CircleCI provides two environment variables
that you can use in lieu of the CLI
to configure each container individually.
CIRCLE_NODE_TOTAL
is the total number of
parallel containers being used to run your
job, and CIRCLE_NODE_INDEX
is the index
of the specific container that is
currently running.
See the built-in environment variable documentation
for more details.
Running Split Tests
Globbing and splitting tests does not actually run your tests. To combine test grouping with test execution, consider saving the grouped tests to an environment variable, then passing this variable to your test runner.
TESTFILES=$(circleci tests glob "spec/**/*.rb" | circleci tests split --split-by=timings)
bundle exec rspec -- ${TESTFILES}
The TESTFILES var will have a different value in each container, based on $CIRCLE_NODE_INDEX and $CIRCLE_NODE_TOTAL.
Video: Troubleshooting Globbing
See Also
Other ways to split tests
Test suite split with Knapsack Pro for Ruby & JavaScript
3rd Party Library or Application Notice: Knapsack Pro
Knapsack Pro is not developed or supported by CircleCI. Please check with the owner if you have issues using it with CircleCI. If you're unable to resolve the issue you can search and ask on our forum, Discuss.
You can allocate tests in a dynamic way across parallel CI nodes with Knapsack Pro Queue Mode. This way you can auto balance CI nodes timing affected by the randomness of tests time execution to run faster CI builds. See CI build time graph examples.