cmd+/

Go to Application

Getting started with Smarter Testing Beta

16 days ago · 17 min read

Cloud

Smarter Testing is available in beta. This means the product is in early stages and you may encounter bugs, unexpected behavior, or incomplete features. When the feature is made generally available, there will be a cost associated with access and usage.

Refer to our Discuss post for more information about our beta launch.

Smarter Testing reduces test execution time while maintaining test confidence. It provides three independent features you can enable incrementally:

Test impact analysis — run only the tests impacted by your code changes.
Dynamic test splitting — evenly distribute tests across parallel execution nodes.
Auto rerun failed tests — automatically retry failed tests.

This guide walks you through the foundational setup that all three features build on:

Install the testsuite CLI plugin locally.
Configure a .circleci/test-suites.yml file to discover and run your tests.
Validate and run your tests using the testsuite command.

Before you begin

You need a repository with an existing test suite. The setup process guides you through configuring JUnit XML output and installing the CLI.

You can complete steps 1 through 3 (install, configure, and validate locally) without a CircleCI account. A CircleCI account with a project connected to your VCS is only required for step 4, when you run tests in CI.

Terminology

The following terms are used throughout the Smarter Testing documentation:

Test atom: The smallest unit of work your test runner can execute independently - typically a test file (for example, tests/auth_test.py), but can also be a test package or module.
Test suite: A named configuration in .circleci/test-suites.yml that defines how to discover and run your test atoms.

1. Install the testsuite plugin locally

The testsuite plugin can be used both locally and in CI. Install locally to validate your commands work immediately, without waiting for them to run in CI.

$ brew install circleci/tap/circleci-testsuite

Verify with:

$ sha256sum -c <(echo "6c63d0e393ace6163b1af2c79deefb8bfecb565b7fcfd2af011d6c12208291f6  ./circleci-testsuite.gz")

After downloading the plugin and verifying the checksum, complete the following steps:

Decompress the binary.
```
$ gunzip circleci-testsuite.gz
```
Set the execute bit.
```
$ chmod u+x circleci-testsuite
```
Move it to a directory on your shell’s $PATH.
```
$ mv circleci-testsuite "${HOME}/bin/"
```

Verify with:

$ sha256sum -c <(echo "a8cda080448c7ccac09ecb7ed988405f658d3717ca1bc4ee799a86fb03f84c99  ./circleci-testsuite.gz")

After downloading the plugin and verifying the checksum, complete the following steps:

Decompress the binary.
```
$ gunzip circleci-testsuite.gz
```
Set the execute bit.
```
$ chmod u+x circleci-testsuite
```
Move it to a directory on your shell’s $PATH.
```
$ mv circleci-testsuite "${HOME}/bin/"
```

Verify with:

$ sha256sum -c <(echo "13d187e3c708b6fba124b69c2677b7621f1456e2603b8db84c86e5a000033431  ./circleci-testsuite.gz")

After downloading the plugin and verifying the checksum, complete the following steps:

Decompress the binary.
```
$ gunzip circleci-testsuite.gz
```
Set the execute bit.
```
$ chmod u+x circleci-testsuite
```
Move it to a directory on your shell’s $PATH.
```
$ mv circleci-testsuite "${HOME}/bin/"
```

Verify with:

$ sha256sum -c <(echo "be60f126e9fce1a578bcc72ddcbe7ebecce5644c61ef77be63a0969b9d9ae6fc  ./circleci-testsuite.gz")

After downloading the plugin and verifying the checksum, complete the following steps:

Decompress the binary.
```
$ gunzip circleci-testsuite.gz
```
Set the execute bit.
```
$ chmod u+x circleci-testsuite
```
Move it to a directory on your shell’s $PATH.
```
$ mv circleci-testsuite "${HOME}/bin/"
```

Verify with:

$ sha256sum -c <(echo "ac61daa2f70d4e1ac8da968dd28aa8d1c09c11cfc9ee4c13d88a23f18ae0a063  ./circleci-testsuite.gz")

After downloading the plugin and verifying the checksum, complete the following steps:

Decompress the binary.
```
$ gunzip circleci-testsuite.gz
```
Set the execute bit.
```
$ chmod u+x circleci-testsuite
```
Move it to a directory on your shell’s $PATH.
```
$ mv circleci-testsuite "${HOME}/bin/"
```

Verify with:

$ sha256sum -c <(echo "d5c9570cddbb342486aaf0b7851f62080326aea81cda6244c8af1dc854eccb4d  ./circleci-testsuite.exe.gz")

After downloading the plugin and verifying the checksum, complete the following steps:

Decompress the binary.
```
$ gunzip circleci-testsuite.gz
```
Set the execute bit.
```
$ chmod u+x circleci-testsuite
```
Move it to a directory on your shell’s $PATH.
```
$ mv circleci-testsuite "${HOME}/bin/"
```

2. Add a `.circleci/test-suites.yml` file at your project root

Create a .circleci/test-suites.yml file in your project. Choose the starter configuration for your test runner below and copy it into the file.

The configuration uses two template variables that Smarter Testing replaces at runtime:

<< test.atoms >> — replaced with the list of test files to run.
<< outputs.junit >> — replaced with the file path for JUnit XML results.

---
name: ci tests
discover: vitest list --filesOnly
run: vitest run --reporter=junit --outputFile="<< outputs.junit >>" --bail 0 << test.atoms >>
outputs:
  junit: test-reports/tests.xml

---
name: ci tests
discover: jest --listTests
run: JEST_JUNIT_OUTPUT_FILE="<< outputs.junit >>" jest --runInBand --reporters=jest-junit --bail << test.atoms >>
outputs:
  junit: test-reports/tests.xml

---
name: ci tests
discover: find ./tests -type f -name '*_test.js'
run: mocha --reporter xunit --reporter-options output="<< outputs.junit >>",showRelativePaths << test.atoms >>
outputs:
  junit: test-reports/tests.xml

---
name: ci tests
discover: find ./tests -type f -name 'test*.py'
run: pytest --disable-pytest-warnings --no-header --quiet --tb=short --junit-xml="<< outputs.junit >>" << test.atoms >>
outputs:
  junit: test-reports/tests.xml

---
name: ci tests
discover: go list -f '{{ if or (len .TestGoFiles) (len .XTestGoFiles) }} {{ .ImportPath }} {{end}}' ./...
run: go tool gotestsum --junitfile="<< outputs.junit >>" -- -race -count=1 << test.atoms >>
outputs:
  junit: test-reports/tests.xml

Install the RSpec JUnit Formatter to collect JUnit XML test reports.

$ gem install rspec_junit_formatter

---
name: ci tests
discover: find spec -type f -name '*_spec.rb'
run: bundle exec rspec --format RspecJunitFormatter --out "<< outputs.junit >>" << test.atoms >>
outputs:
  junit: test-reports/tests.xml

Smarter Testing is test runner agnostic. Replace the discover and run examples below with your test runner commands. The next step validates that the commands are set up correctly.

---
name: ci tests
# Replace with the command that lists your tests
discover: my-test-runner --only-list-tests
# Replace with the command that runs your tests, be sure to enable JUnit and
# send the JUnit output to the '<< outputs.junit >>' template variable.
run: my-test-runner --junit=<< outputs.junit >> --run << test.atoms >>
outputs:
  junit: test-reports/tests.xml

3. Validate and run locally

Use the --doctor flag to validate that your test-suites.yml is set up correctly. The CLI runs through a series of checks, executing tests and validating the output. If any results look incorrect, an action item is provided to resolve it.

$ circleci run testsuite "ci tests" --doctor

Follow the steps until all checks pass.

Run the circleci run testsuite CLI tooling from the directory where your tests are located. This is typically your repository root, but for monorepos it can be the root of a subpackage (for example, cd service-1 && circleci run testsuite "ci tests").

The testsuite will automatically find the .circleci/test-suites.yml configuration file by walking up the directory tree. All commands (discover, run, analysis) execute relative to where you run the CLI, so avoid using cd or --directory flags within your commands.

Once all checks pass, run all tests locally by removing the --doctor flag:

$ circleci run testsuite "ci tests"

4. Run in CI

When adding the testsuite command to your CircleCI jobs, there is no need to install the testsuite plugin as it is already available in CircleCI Docker containers.

In your .circleci/config.yml, replace your existing test command with circleci run testsuite — the same command you used locally. Keep all other job steps the same.

Verify that store_test_results points to the directory that matches outputs.junit in your test-suites.yml file.

Example CircleCI test job configuration with the testsuite command

version: 2.1
jobs:
  test:
    executor: node-with-service
    steps:
      - setup
      # Replace previous tests command e.g. vitest run --reporter=junit ...
      - run: circleci run testsuite "ci tests"
      - store_test_results:
          # This directory must match the directory of `outputs.junit` in your
          # test-suites.yml
          path: test-reports

Commit both .circleci/test-suites.yml and .circleci/config.yml to your feature branch and push to your VCS.

Verify in CI

After your pipeline runs, verify the test job behaves as expected before merging:

In the CircleCI web app, navigate to your pipeline and open the test job.
Check the Tests tab and confirm the number of test results matches what you see when running tests without the testsuite command. A difference in test results could indicate a misconfiguration in your discover command.
Compare the job duration to a recent run without the testsuite command. The runtime is expected to be similar. A significant difference may indicate a misconfiguration in your test-suites.yml.

Next steps

All of your tests now run locally and in CI using the testsuite command. Next, enable the Smarter Testing features that fit your needs. Each feature can be enabled independently:

Set up Test Impact Analysis to run only impacted tests based on code changes.
Use Dynamic Test Splitting to evenly split tests across parallel nodes.
Auto Rerun Failed Tests to automatically retry flaky tests.