Test splitting and parallelism
Use parallelism and test splitting to:
- Reduce the time taken for the testing portion of your CI/CD pipeline.
- Specify a number of executors across which to split your tests.
- Split your test suite using one of the options provided by the CircleCI CLI: by name, size or by using timing data.
If you are interested to read about concurrent job runs, see the Concurrency overview page.
Introduction
Pipelines are often configured so that a set of tests are run each time code is committed. The more tests your project has, the longer it will take for them to complete using a single compute resource. To reduce this time, you can split your tests and run them across multiple, parallel-running execution environments. Test splitting is a great way to speed up the testing portion of your CI/CD pipeline.
CircleCI test splitting lets you intelligently define where splits happen across a test suite:
- By name
- By size
- Using timing data
It is also possible to use the CLI to manually allocate tests across parallel environments. Another alternative is to use environment variables instead of the CLI to configure split tests.
Specify a job’s parallelism level
Test suites are conventionally defined at the job level in your .circleci/config.yml
file. The parallelism
key specifies how many independent executors are set up to run the job.
To run a job’s steps in parallel, set the parallelism
key to a value greater than 1
. In the example below, parallelism
is set to 4
, meaning four identical execution environments will be set up for the job.
# ~/.circleci/config.yml
version: 2.1
jobs:
test:
docker:
- image: cimg/base:2022.11
parallelism: 4
With no further changes, the full test
job is still run in each of the four execution environment. In order to automatically run different tests in each environment and reduce the overall time taken to run the tests, you also need to use the circleci tests
CLI commands in your configuration.
Use parallelism with self-hosted runners
To use the parallelism feature with jobs that use self-hosted runners, ensure that you have at least two self-hosted runners associated with the runner resource class that your job will run on. If you set the parallelism value to be greater than the number of active self-hosted runners in a given resource class, the excess parallel tasks that do not have a self-hosted runner on which to execute will queue until a self-hosted runner is available.
For more information, see the Configuration reference page.
How test splitting works
Using timing-based test splitting as an example, timing data from the previous test run is used to split a test suite as evenly as possible over a specified number of test environments running in parallel. This delivers the lowest possible test time for the compute power in use.
Timings-based test splitting gives the most accurate split, and is guaranteed to optimize with each test suite run. The most recent timings data is always used to define where splits are made.
As an example, take a Go test suite. Here, all tests run sequentially in a single test environment, a Docker container:
jobs:
build:
docker:
- image: cimg/go:1.18.1
resource_class: large
working_directory: ~/my-app
steps:
- run: go test
To split these tests using timing data:
- Introduce parallelism to spin up a number of identical test environments (4 in this example)
- Use the
circleci tests split
command, with the--split-by=timings
flag to split the tests evenly across all executors.
jobs:
build:
docker:
- image: cimg/go:1.18.1
parallelism: 4
resource_class: large
working_directory: ~/my-app
steps:
- run: go test -v $(go list ./... | circleci tests split --split-by=timings)
For a more detailed walkthrough, read the guide to using the CLI to split tests, or follow our Test splitting tutorial.
Manual allocation
For full control over how tests are split across parallel executors, CircleCI provides two environment variables that you can use in place of the CLI to configure each container individually.
$CIRCLE_NODE_TOTAL
is the total number of parallel containers being used to run your job.$CIRCLE_NODE_INDEX
is the index of the specific container that is currently running.
The CLI looks up the number of available execution environments ($CIRCLE_NODE_TOTAL
), along with the current container index ($CIRCLE_NODE_INDEX
). Then, it uses deterministic splitting algorithms to split the test files across all available containers.
The number of containers is specified by the parallelism
key in the project configuration file.
The current container index is automatically picked up from the $CIRCLE_NODE_INDEX
environment variable, but can be manually set by using the --index
flag.
circleci tests split --index=0 test_filenames.txt
Refer to the Project values and variables page for more details.
Other ways to split tests
Some third party applications and libraries might help you to split your test suite. These applications are not developed or supported by CircleCI. Please check with the owner if you have issues using it with CircleCI. If you are unable to resolve the issue you can search and ask on our forum, Discuss.
-
Knapsack Pro - Enables allocating tests dynamically across parallel CI nodes, allowing your test suite execution to run faster. See CI build time graph examples.
- phpunit-finder - This is a helper CLI tool that queries
phpunit.xml
files to get a list of test filenames and print them. This is useful if you want to split tests to run them in parallel based on timings on CI tools. -
go list - Use the built-in Go command
go list ./...
to glob Golang packages. This allows splitting package tests across multiple containers.go test -v $(go list ./... | circleci tests split)
-
Playwright - This is a framework for web testing and automation and allows running sharded tests out of the box. For more details see playwright docs.
job-name: executor: pw-focal-development parallelism: 4 steps: - run: SHARD="$((${CIRCLE_NODE_INDEX}+1))"; npx playwright test -- --shard=${SHARD}/${CIRCLE_NODE_TOTAL}
Next steps
Help make this document better
This guide, as well as the rest of our docs, are open source and available on GitHub. We welcome your contributions.
- Suggest an edit to this page (please read the contributing guide first).
- To report a problem in the documentation, or to submit feedback and comments, please open an issue on GitHub.
- CircleCI is always seeking ways to improve your experience with our platform. If you would like to share feedback, please join our research community.
Need support?
Our support engineers are available to help with service issues, billing, or account related questions, and can help troubleshoot build configurations. Contact our support engineers by opening a ticket.
You can also visit our support site to find support articles, community forums, and training resources.
CircleCI Documentation by CircleCI is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.