Use dynamic test splitting Preview
9 days ago
| Smarter Testing is available in preview. This means the product is in early stages and you may encounter bugs, unexpected behavior, or incomplete features. When the feature is made generally available, there will be a cost associated with access and usage. |
Dynamic test splitting distributes tests across parallel nodes to minimize node idle time. Unlike static splitting, which divides tests evenly upfront, dynamic splitting uses a shared queue that nodes pull from continuously. This ensures balanced workloads even when some nodes start late or run slower than expected.
How it works
When you configure parallelism in your job and enable dynamic test splitting, Smarter Testing automatically:
-
Sorts tests by duration by retrieving timing data from previous runs, with longer tests prioritized first.
-
Creates a shared queue containing all tests to be run.
-
Enables nodes to pull dynamically. Each parallel node continuously pulls test batches from the queue.
-
Adjusts batch sizes. Larger batches at the start, smaller batches as the queue empties to ensure even distribution.
Dynamic test splitting prevents any single slow node from becoming a bottleneck. Even if some nodes start late or run slower than expected, this dynamic batching ensures work stays balanced across all available nodes.
Prerequisites
Before enabling dynamic test splitting, ensure you have completed the Getting Started With Smarter Testing guide and have:
-
Installed the
testsuiteCLI plugin -
Configured your
.circleci/test-suites.ymlwithdiscoverandruncommands -
Verified your tests run successfully with the
testsuitecommand
Configure dynamic test splitting
1. Configure parallelism in your CircleCI job
Update your .circleci/config.yml to add parallelism to your test job:
version: 2.1
jobs:
test:
executor: node-with-service
parallelism: 4 # Run tests across 4 parallel nodes
steps:
- checkout
- restore_cache:
keys:
- deps-{{ checksum "package-lock.json" }}
- run: npm ci
- run: circleci run testsuite "ci tests"
- store_test_results:
path: test-reportsWithout dynamic test splitting enabled, Smarter Testing will default to use a static test splitting approach, which pre-distributes tests to nodes at the start of the job based on timing data.
2. Store test results
Test distribution (both static and dynamic) uses timing data from JUnit XML files to optimize how tests are distributed. Make sure you are storing test results by running the following steps:
# .circleci/config.yml
jobs:
test:
steps:
- run: circleci run testsuite "ci tests"
- store_test_results:
path: test-reports # Must match outputs.junit directoryAnd that your test suite configuration specifies the JUnit output location:
# .circleci/test-suites.yml
---
name: ci tests
discover: vitest list --filesOnly
run: vitest run --reporter=junit --outputFile="<< outputs.junit >>" --bail 0 << test.atoms >>
outputs:
junit: test-reports/tests.xml3. Enable dynamic test splitting
To enable dynamic test splitting instead of the default static approach, set dynamic-test-splitting: true in your test suite configuration:
# .circleci/test-suites.yml
---
name: ci tests
discover: vitest list --filesOnly
run: vitest run --reporter=junit --outputFile="<< outputs.junit >>" --bail 0 << test.atoms >>
outputs:
junit: test-reports/tests.xml
options:
dynamic-test-splitting: trueWith dynamic test splitting enabled, tests are distributed via a shared queue that nodes pull from continuously, rather than being pre-assigned at the start.
Commit and push this change to verify dynamic test splitting is working in CI.
At this point, you have successfully set up dynamic test splitting. Next, you can enable other features of Smarter Testing:
-
Enable Auto Rerun to handle flaky tests.
Or continue learning more about dynamic test splitting below.
Troubleshooting
Tests are running slower with dynamic test splitting enabled
Dynamic test splitting pulls multiple batches from the queue, requiring the test runner to start up for each batch. If your test runner has significant startup overhead, these repeated startups can outweigh the benefits of dynamic test splitting. Disable dynamic test splitting to use static test splitting instead, where each node starts up once and runs all its pre-assigned tests in a single batch.
Tests not being split evenly across nodes
Symptoms: Some parallel nodes finish much faster than others, or tests are not distributed evenly.
Solution: Verify that your test suite configuration includes historical timing data and that all test files are being detected. Check the step output for the "Sorted X tests" or "Autodetected timing" messages to confirm that test atoms are being sorted by timing.
Debugging steps:
-
Check that all test atoms are discovered with the discover command.
-
Verify parallelism is set correctly in your
.circleci/config.yml. -
Ensure test results are being stored with
store_test_results- this is how timing data is collected. -
Run the job multiple times to allow timing data to accumulate.
Dynamic test splitting configuration options
The following options are available to be defined in the options map in test-suites.yml config:
| Options Field | Default | Description |
|---|---|---|
|
|
Whether tests should be distributed across a shared queue with dynamic batching. When |
|
|
The time in minutes a node will wait for tests to become available from the queue when running in parallel. |