CircleCI Smarter Testing: Stop running tests that don't matter

Testing consumes up to 50% of CI/CD pipeline time. That’s the single most expensive place in your delivery pipeline, and a meaningful share of that time is often wasted running tests unaffected by the change.

Every time a developer pushes a commit, the full test suite runs. A two-line bug fix in auth.py triggers 2,000 tests, including 1,900 that don’t touch anything near auth.py. Those unrelated tests are overwhelmingly likely to pass, but the pipeline still spends 20 minutes running them. In the meantime, the developer context-switches to something else, making it harder to pick the original task back up when results arrive.

AI is making this worse. Developers can now generate code and write tests faster than ever. But faster code creation means more commits, more tests, and more pressure on a pipeline that was already running at capacity. When your test suite doubles in 12 months but your pipeline doesn’t get any smarter, you’ve just doubled the time developers spend waiting.

Smarter Testing is one of the ways CircleCI helps teams reclaim that time. It’s a suite of three features, with more to come, that reduce test execution time without sacrificing confidence in your results—built natively into CircleCI, requiring no new vendors, no new contracts, and no new infrastructure. Early users are seeing up to 4x faster test runs.

Why this problem is more expensive than it looks

Slow CI costs more than pipeline runtime. The bigger cost is the hit to developer productivity while tests run.

Research shows it takes an average of 23 minutes to fully regain focus after an interruption. A 30-minute test run doesn’t cost 30 minutes of productivity; it costs an entire working session. Multiply that by 4 commits per developer per day across a team of 50, and you’re looking at a significant fraction of your engineering capacity consumed by waiting.

Slow CI also makes it harder for teams to maintain good CI hygiene: smaller, more frequent commits, continuous validation, and fast integration back to main. These practices reduce risk, improve review quality, and accelerate delivery, but teams can’t build that habit if the feedback loop that would reinforce it is too slow to close.

The common response—buying more compute—addresses the symptom without fixing the cause. Upgrading to larger resource classes makes every pipeline run more expensive, forever. Smarter Testing makes the existing compute work smarter: fewer tests run, nodes are better balanced, and flaky tests don’t trigger full reruns.

At Clazar, a backend suite of roughly 4,000 tests was adding 12–15 minutes to every PR. After enabling Smarter Testing, feature branch runs dropped to 3–5 minutes. At Liti, PR jobs are now 4x faster.

How Smarter Testing works

Smarter Testing includes three features designed to eliminate unnecessary test work. You can use them independently or combine them, depending on where your test pipeline is wasting time.

1. Intelligent Test Selection: run only what matters

The biggest time savings comes from not running tests that can’t possibly be affected by your changes. Intelligent Test Selection (ITS), powered by Test Impact Analysis (TIA), does exactly that.

Our approach uses code coverage data to map how tests interact with your source code. To guarantee safety and prevent under-selection, the testsuite command starts with your full test suite selected, then conservatively deselects only the tests proven to be unaffected by modified files.

This isn’t sampling or heuristic guessing. If your change touches auth.py, only the tests that cover auth.py (and code it calls) are selected. Tests for unrelated modules are skipped. Your full suite still runs on your main branch to maintain the coverage baseline, but on feature branches—where fast feedback matters most—you can routinely skip 60–90%+ of your tests.

When ITS is the right fit: ITS has the highest potential impact for codebases where most changes are localized. If you have a large test suite and most PRs touch a narrow area of the codebase, ITS will skip the overwhelming majority of tests on every feature branch run. It’s particularly valuable in AI-assisted development workflows, where developers iterate rapidly and need tight feedback loops to stay in flow.

ITS requires your test runner to generate code coverage data. Most modern frameworks support this (Jest, pytest, Go test, Vitest, RSpec, Mocha, Cypress, and others). No changes to your tests are needed. Just configure coverage output and let TIA do the rest.

On test confidence: Because TIA is based on actual coverage data rather than guesswork, it’s safe to rely on. And because Smarter Testing records which tests ran and why, you get more visibility into your test suite than you had before, not less. The Test Analytics dashboard shows skip rates, run time trends, and per-test history, so you can see exactly what ran on any given commit and what was skipped.

2. Dynamic Test Splitting: eliminate idle time

If you’re using parallelism to run tests across multiple nodes, you’ve likely noticed the nodes don’t always finish together. One node wraps up early while another is still grinding through a long test file. Your total job time is determined by the slowest node—so that idle time is pure waste.

Static test splitting assigns tests to nodes upfront based on historical JUnit timing data. The problem is that JUnit timings often don’t reflect real wall-clock runtime—they miss fixed costs like runner startup, per-batch setup overhead, and hook execution. The result is uneven splits even when reported durations look balanced.

Dynamic Test Splitting takes a different approach. Instead of pre-assigning tests, it maintains a shared queue that nodes pull from continuously. It learns the gap between reported and actual runtime over time, accounting for fixed costs and per-test overhead. Nodes that start late or run slower receive smaller batches so they still finish at roughly the same time.

When dynamic splitting is the right fit: Dynamic splitting helps most when parallel nodes finish at noticeably different times because of uneven test durations, startup delays, or inaccurate timing data. If your nodes already finish within seconds of one another, the benefit will be limited.

3. Auto Rerun Failed Tests: handle flakiness automatically

Flaky tests are nearly universal in active test suites—tests that fail intermittently due to timing issues, external service calls, or environment variance. The traditional response is to wait for a CI failure, manually identify which tests failed, navigate to the CircleCI UI, and trigger a rerun. This is tedious, slow, and entirely manual.

Auto Rerun Failed Tests automates recovery. When a test fails, Smarter Testing immediately retries it, no human intervention required. The process repeats until all tests pass or the maximum number of retries is reached. If all tests eventually pass, the errors are suppressed and the job succeeds, keeping pipelines moving without noise or manual triage.

You control retry behavior via two settings in test-suites.yml:

• max-auto-rerun: the maximum number of retry attempts per failing test atom (0–10)
• auto-rerun-duration: a time limit or percentage of total runtime to spend on retries (e.g., 5m, 20%)

Both can be set simultaneously; retries stop when either condition is reached.

When auto rerun is the right fit: If your test suite has known flaky tests, this is an immediate improvement. It’s especially useful alongside Intelligent Test Selection. When you’re running fewer tests overall, even occasional flakiness in a small subset can cause false failures. Auto rerun catches those without requiring manual intervention.

Note that auto rerun handles flakiness, not real bugs. If a test fails because of an actual code problem, it will keep failing on retry and the job will ultimately fail—which is correct behavior. The goal is to stop treating intermittent environmental failures as blocking issues.

What’s included: free vs. paid

When you adopt the circleci testsuite run command you get a free tier that includes everything the legacy circleci tests run command offered: test running, static test splitting across parallel nodes, and rerunning only failed tests. If you migrate to testsuite and use only these capabilities, there’s no additional cost.

On top of that foundation, Smarter Testing’s paid features go further: Intelligent Test Selection (powered by Test Impact Analysis), Dynamic Test Splitting, Auto Rerun Failed Tests, and the Test Analytics dashboard. These are charged based on stored test results per billing period, with generous free thresholds included in every plan tier.

Setting up Smarter Testing

All three features build on the same foundation: a .circleci/test-suites.yml file that defines how your tests are discovered and run, and the circleci testsuite run command that replaces your existing test command in CI. You configure this once, then enable individual features by adding options to the YAML file.

The CircleCI Testing Wizard is a self-driving CLI tool that handles the foundational Smarter Testing setup for you: installing the CLI, running the validation checks, and handing off to Claude when something needs fixing.

Step 1: Run the doctor command

Ensure you have the latest CircleCI CLI installed. Then run the doctor command from the directory where your tests are located (typically your repository root):

$ circleci testsuite doctor "ci tests"

The CLI walks you through a series of checks, executes your tests, validates the output, and provides specific action items for anything that looks misconfigured. Follow the steps until all checks pass.

Monorepos: Run the CLI from each subpackage root, not the repository root. For example: cd service-1 && circleci testsuite doctor "ci tests". The CLI looks for .circleci/test-suites.yml relative to where it’s invoked.

You can also run the doctor command as a CI job step to validate in your actual CI environment:

version: 2.1
jobs:
  doctor:
    docker:
      - image: cimg/node:20.0
    steps:
      - checkout
      - run: circleci testsuite doctor "ci tests"

Step 2: Replace your test command

Once the doctor command passes, replace your existing test command in .circleci/config.yml with circleci testsuite run "ci tests".

Before:

version: 2.1
jobs:
  test:
    docker:
      - image: cimg/node:20.0
    steps:
      - checkout
      - run: npm test
      - store_test_results:
          path: test-reports

After:

version: 2.1
jobs:
  test:
    docker:
      - image: cimg/node:20.0
    steps:
      - checkout
      - run: circleci testsuite run "ci tests"

Step 3: Enable the features you want

Features are enabled via the options block in .circleci/test-suites.yml. A minimal configuration, storing tests, looks like this:

---
name: ci tests
discover: jest --listTests
run: JEST_JUNIT_OUTPUT_FILE="<< outputs.junit >>" jest --runInBand --reporters=jest-junit --bail << test.atoms >>
options:
  store-test-results: true

Then add the options for whichever features you want:

Intelligent Test Selection (paid):

options:
  test-impact-analysis: true

Dynamic Test Splitting (paid, requires parallelism > 1 in your job config):

options:
  dynamic-test-splitting: true

Auto Rerun Failed Tests (paid):

options:
  max-auto-rerun: 3
  # Optional: also limit by time
  auto-rerun-duration: 5m

You can combine all three:

---
name: ci tests
discover: jest --listTests
run: JEST_JUNIT_OUTPUT_FILE="<< outputs.junit >>" jest --runInBand --reporters=jest-junit --bail << test.atoms >>
options:
  test-impact-analysis: true
  dynamic-test-splitting: true
  max-auto-rerun: 3
  store-test-results: true

Step 4: Verify locally and in CI

Locally

$ circleci testsuite run "ci tests"

In CI

Commit both .circleci/test-suites.yml and .circleci/config.yml to a feature branch and push. After the pipeline runs:

• Open the Tests tab and confirm the test count matches what you’d expect. A significant mismatch may indicate a misconfiguration in your discover command.
• Compare the job duration to a recent run without testsuite. It should be similar at baseline. Speedups from Intelligent Test Selection appear on subsequent runs as the impact map is established.
• For dynamic splitting, check the Timings tab to confirm tests are distributed more evenly across nodes.

The ROI case

For teams deciding whether the paid tier is worth it, the math is straightforward. Consider a 100-developer team running a 20-minute test suite, with each developer pushing 4 feature branch commits per day:

Without Smarter Testing, that’s 400 feature branch runs × 20 minutes = 8,000 minutes of pipeline time per day. With Intelligent Test Selection reducing feature branch test volume by 60%, the average run drops to ~8 minutes—3,200 minutes per day. Over a month, that’s 105,600 minutes of compute savings, plus every developer recovering roughly 12 minutes per commit.

At a fully-loaded developer cost of $135/hour, recovering even 25% of that wait time as productive work translates to tens of thousands of dollars in recovered engineering capacity each month. The Smarter Testing fee, based on stored test results, is typically a rounding error compared to that figure.

The ROI is driven almost entirely by developer time. Compute savings matter, but the larger benefit is getting useful feedback to developers faster so they can keep changes moving through the pipeline.

Troubleshooting common issues

Test count in the Tests tab doesn’t match expected results

This usually means the discover command in test-suites.yml isn’t finding all test files, or is picking up files it shouldn’t. Re-run circleci testsuite doctor "ci tests" and look for mismatches in the discovery output. Verify your glob patterns and that the CLI is being invoked from the correct directory.

First run after enabling Intelligent Test Selection still runs all tests

Expected behavior. ITS needs a full coverage baseline before it can determine which tests are impacted by a change. The first run on your main branch establishes that baseline. After that, subsequent feature branch runs start selecting only impacted tests.

Dynamic test splitting doesn’t improve node balance

Dynamic splitting improves as it accumulates timing data over multiple runs. Early runs may look similar to static splitting. If nodes are still unbalanced after many runs, check whether your test runner startup time is high relative to test runtime—if startup takes longer than the tests themselves, splitting across multiple batches can actually increase total runtime. Dynamic splitting works best when test runner startup is fast relative to test execution.

Auto rerun keeps retrying the same test without passing

A test that fails consistently is not flaky—it’s broken. Auto rerun will exhaust the max-auto-rerun limit and the job will correctly fail. Increase your retry limit if the test is genuinely intermittent but needs more attempts, or add auto-rerun-duration to bound the time spent retrying.

circleci testsuite command not found

Ensure you have the latest CircleCI CLI installed. Run circleci update to upgrade, or reinstall from cli.circleci.com. The testsuite subcommand requires a recent CLI version.

Tests pass locally but fail in CI

Run circleci testsuite doctor "ci tests" as a CI job step to validate the configuration in your actual CI environment. Differences in working directory, available tools, or coverage configuration can cause the testsuite command to behave differently locally vs. in CI. The doctor command surfaces those differences.

Coverage data isn’t being picked up for Intelligent Test Selection

TIA requires your test runner to produce code coverage output. Verify your test command includes the appropriate coverage flag (e.g., --coverage for Jest, --cov for pytest) and that the output path is configured correctly in test-suites.yml. Re-run the doctor command. It will flag missing or misconfigured coverage output specifically.

Monorepo: tests aren’t being discovered correctly

Make sure you’re running the CLI from the subpackage root, not the repository root. The testsuite command executes relative to where it’s invoked. Avoid using cd commands or --directory flags within your discover or run commands. Set the working directory in the job step instead.

Getting started

Running fewer unnecessary tests means faster feedback without lowering confidence in the result. Smarter Testing gives teams a few different ways to get there, depending on where their test pipeline is losing time.

Start by adopting the circleci testsuite run command (free, full parity with the legacy command) and verifying your test setup. Then enable the paid features that address your biggest bottlenecks. Most teams see the largest immediate gains from Intelligent Test Selection, and the benefits increase as Dynamic Test Splitting and Auto Rerun Failed Tests eliminate additional sources of wasted test time.

Further reading:

• Getting started with Smarter Testing
• Set up Test Impact Analysis
• Use Dynamic Test Splitting
• Auto Rerun Failed Tests
• Test Suite Configuration Reference