Test splitting and parallelism

Use parallelism and test splitting to:

• Reduce the time taken for the testing portion of your CI/CD pipeline.
• Specify a number of executors across which to split your tests.
• Split your test suite using one of the options provided by the CircleCI CLI: by name, size or by using timing data.

If you are interested to read about concurrent job runs, see the Concurrency overview page.

Pipelines are often configured so that a set of tests are run each time code is committed. The more tests your project has, the longer it will take for them to complete using a single compute resource. To reduce this time, you can split your tests and run them across multiple, parallel-running execution environments. Test splitting is a great way to speed up the testing portion of your CI/CD pipeline.

CircleCI test splitting lets you intelligently define where splits happen across a test suite:

• By name
• By size
• Using timing data

It is also possible to use the CLI to manually allocate tests across parallel environments. Another alternative is to use environment variables instead of the CLI to configure split tests.

Test suites are conventionally defined at the job level in your .circleci/config.yml file. The parallelism key specifies how many independent executors are set up to run the job.

To run a job’s steps in parallel, set the parallelism key to a value greater than 1. In the example below, parallelism is set to 4, meaning four identical execution environments will be set up for the job.

# ~/.circleci/config.yml
version: 2.1
jobs:
  test:
    docker:
      - image: cimg/base:2022.11
    parallelism: 4

With no further changes, the full test job is still run in each of the four execution environment. In order to automatically run different tests in each environment and reduce the overall time taken to run the tests, you also need to use the circleci tests CLI commands in your configuration.

To use the parallelism feature with jobs that use self-hosted runners, ensure that you have at least two self-hosted runners associated with the runner resource class that your job will run on. If you set the parallelism value to be greater than the number of active self-hosted runners in a given resource class, the excess parallel tasks that do not have a self-hosted runner on which to execute will queue until a self-hosted runner is available.

For more information, see the Configuration reference page.

Using timing-based test splitting as an example, timing data from the previous test run is used to split a test suite as evenly as possible over a specified number of test environments running in parallel. This delivers the lowest possible test time for the compute power in use.

Timings-based test splitting gives the most accurate split, and is guaranteed to optimize with each test suite run. The most recent timings data is always used to define where splits are made.

As an example, take a Go test suite. Here, all tests run sequentially in a single test environment, a Docker container:

jobs:
  build:
    docker:
      - image: cimg/go:1.18.1
    resource_class: large
    working_directory: ~/my-app
    steps:
      - run: go test

To split these tests using timing data:

1. Introduce parallelism to spin up a number of identical test environments (4 in this example)
2. Use the circleci tests split command, with the --split-by=timings flag to split the tests evenly across all executors.

jobs:
  build:
    docker:
      - image: cimg/go:1.18.1
    parallelism: 4
    resource_class: large
    working_directory: ~/my-app
    steps:
      - run: go test -v $(go list ./... | circleci tests split --split-by=timings)

For a more detailed walkthrough, read the guide to using the CLI to split tests, or follow our Test splitting tutorial.

For full control over how tests are split across parallel executors, CircleCI provides two environment variables that you can use in place of the CLI to configure each container individually.

• $CIRCLE_NODE_TOTAL is the total number of parallel containers being used to run your job.
• $CIRCLE_NODE_INDEX is the index of the specific container that is currently running.

The CLI looks up the number of available execution environments ($CIRCLE_NODE_TOTAL), along with the current container index ($CIRCLE_NODE_INDEX). Then, it uses deterministic splitting algorithms to split the test files across all available containers.

The number of containers is specified by the parallelism key in the project configuration file.

The current container index is automatically picked up from the $CIRCLE_NODE_INDEX environment variable, but can be manually set by using the --index flag.

circleci tests split --index=0 test_filenames.txt

Refer to the Project values and variables page for more details.

Some third party applications and libraries might help you to split your test suite. These applications are not developed or supported by CircleCI. Please check with the owner if you have issues using it with CircleCI. If you are unable to resolve the issue you can search and ask on our forum, Discuss.

• Knapsack Pro - Enables allocating tests dynamically across parallel CI nodes, allowing your test suite execution to run faster. See CI build time graph examples.

• phpunit-finder - This is a helper CLI tool that queries phpunit.xml files to get a list of test filenames and print them. This is useful if you want to split tests to run them in parallel based on timings on CI tools.

• go list - Use the built-in Go command go list ./... to glob Golang packages. This allows splitting package tests across multiple containers.

  go test -v $(go list ./... | circleci tests split)
  

• Playwright - This is a framework for web testing and automation and allows running sharded tests out of the box. For more details see playwright docs.

  job-name:
    executor: pw-focal-development
    parallelism: 4
    steps:
      - run: SHARD="$((${CIRCLE_NODE_INDEX}+1))"; npx playwright test -- --shard=${SHARD}/${CIRCLE_NODE_TOTAL}
  

• Use the CircleCI CLI to split tests
• Test splitting to speed up your pipelines
• Troubleshooting Test Splitting
• Collecting Test Data
• Test Insights