cmd+/

Set up test impact analysis Beta

Cloud

Smarter Testing is available in beta. This means the product is in early stages and you may encounter bugs, unexpected behavior, or incomplete features. When the feature is made generally available, there will be a cost associated with access and usage.

Refer to our Discuss post for more information about our beta launch.

Test impact analysis speeds up CI by running only the tests affected by your code changes. CircleCI tracks which tests exercise which source files, then skips tests that cover unchanged source files.

Is my project a good fit for test impact analysis?

This section outlines several project features that indicate your project is a good candidate for test impact analysis:

Built-in coverage support

Test impact analysis relies on code coverage data to determine which tests affect which files. Frameworks with built-in coverage support - Jest, pytest, Go test, Vitest and RSpec - make this straightforward.

If your framework has no native coverage support, generating the data test impact analysis needs may require significant rework of your test setup.

Tests run in the same process

Coverage data works best when tests directly import and run the source code in the same process.

When tests call code running in separate containers or services, collecting and consolidating coverage data across those service boundaries is not straightforward.

How it works

Test impact analysis is split into two phases:

Analysis runs on your default branch with coverage instrumentation, producing impact data. Impact data describes the relationship between files and tests.
Selection runs on feature branches, using the impact data to choose which tests to run based on what changed.

Analysis and selection can be configured to run on any change through CLI flags.

Analysis on default branches

The analysis phase is run on your default branch (or the base branch of a pull request). It runs tests with coverage instrumentation and updates the impact data with the latest test-to-file relationships.

Keeping impact data fresh on the default branch allows the selection phase to select fewer tests without other unrelated changes interfering.

For each file found during coverage analysis, a fast non-crypto hash of its contents is stored in the impact data. The data also tracks which files impact the tests being selected.

Example impact data relationship between a handler and repository.

{
  "version": 1,
  "files": {
    "1": {
      "path": "src/api/handlers.ts",
      "hash": "c9684be83632a628"
    },
    "2": {
      "path": "src/api/handlers.test.ts",
      "hash": "5b8e1a04c7d2f391"
    },
    "3": {
      "path": "src/data/repository.ts",
      "hash": "2e7c4d18a9f6b052"
    },
    "4": {
      "path": "src/data/repository.test.ts",
      "hash": "f1d39e62a08c5b74"
    }
  },
  "edges": {
    "src/api/handlers.test.ts": ["1", "2", "3"],
    "src/data/repository.test.ts": ["3", "4"]
  }
}

files records every file that analysis has seen, along with the hash of its contents at the time it was last analyzed.
edges records, for each test, the IDs of the files that are covered by the test.

The analysis phase typically runs slower than a normal test run because it executes tests with coverage instrumentation. However, this cost pays for itself by enabling the selection phase to skip unaffected tests on subsequent runs.

Selection on feature branches

By default, all tests run on default branches. Test selection runs only affected tests on feature branches, using the impact data produced by analysis on the default branch.

During test selection, the current state of the branch’s checked-out code is compared against the latest impact data. Tests are selected in the following scenarios:

The test is not found in the impact data. This indicates a new test in the checked-out code.
The test failed on the previous run for this branch.
A file affecting the test is not found in the impact data. This indicates the file was removed from the checked-out code.
A file affecting the test hash has changed.

flowchart TD Start([Discovered Test Atoms\nFor each test]) Start --> New{New test?} New -- Yes --> Selected New -- No --> Failed{Previously failed?} Failed -- Yes --> Selected Failed -- No --> Files{Covered files\nchanged or removed?} Files -- Yes --> Selected Files -- No --> Skipped Selected([Selected]) Skipped([Skipped])

The selection rules can be extended to select tests beyond those identified in the impact data.

Prerequisites

Before enabling test impact analysis, ensure you have completed the Getting Started With Smarter Testing guide and have:

Installed the testsuite CLI plugin.
Configured your .circleci/test-suites.yml with discover and run commands.
Verified your tests run successfully with the testsuite command.

1. Enable test impact analysis in your test-suites.yml file

Update your test suite configuration to include an analysis command and the test-impact-analysis option. Keep your existing discover and run commands.

Choose the starter configuration for your test runner and add the analysis line to your .circleci/test-suites.yml.

Follow the instructions to add the Vitest CircleCI Coverage plugin as a dev dependency.