Documentation structure for LLMs (llms.txt)
CircleCI Logo

Testsuite configuration reference Beta

4 hours ago · 21 min read
Cloud

Smarter Testing is available in beta. This means the product is in early stages and you may encounter bugs, unexpected behavior, or incomplete features. When the feature is made generally available, there will be a cost associated with access and usage.

Refer our Discuss post for more information about our beta launch.

This document is a reference for the CircleCI Smarter Testing testsuite configuration keys that are used in the .circleci/test-suites.yml file.

name

The name field is used to run a testsuite with the local plugin and CI subcommand.

Key Required Type Description

name

Y

String

Name of the testsuite to run with the CLI.

Example:

$ circleci run testsuite "frontend tests"
$ circleci run testsuite "backend tests"
---
name: frontend tests
---
name: backend tests

discover

The discover command finds all test atoms for a given test suite. The command’s output is split on both newlines and whitespace to extract individual test atoms.

The discover command should not execute any tests.

Every line of stdout is interpreted as a test atom. Any tool that prints metadata in the stdout must be suppressed.

Key Required Type Description

discover

Y

String

An executable command that can list tests for a test runner, without running them.

Example:

---
name: vitest tests
discover: vitest list --filesOnly
---
name: rspec tests
discover: find spec -type f -name '*_spec.rb'
---
name: go tests
discover: go list ./...

run

The run command executes the test atoms discovered by the discover command using your test runner.

The command needs to be modified to use placeholders:

For test atoms:

  • Use the template variable << test.atoms >> in your run command – this will be replaced with a space-separated list of test atoms to run.

  • If the template variable is not found, the command’s stdin will receive a newline-separated list of test atoms.

For JUnit output:

  • Use the template variable << outputs.junit >> so Smarter Testing can locate your test results.

Key Required Type Description

run

Y

String

A command that executed tests.

Example:

---
name: vitest tests
run: vitest run --reporter=junit --outputFile="<< outputs.junit >>" --silent --bail 0 << test.atoms >>
---
name: rspec tests
run: bundle exec rspec --format RspecJunitFormatter --out "<< outputs.junit >>" << test.atoms >>
---
name: go tests
run: go tool gotestsum --junitfile="<< outputs.junit >>" -- -race -count=1 << test.atoms >>

file-mapper

When test atoms are not files, Smarter Testing cannot determine which test files belong to which test atom. The file-mapper creates a mapping of test atoms to related test files.

  • During analysis, Smarter Testing can associate test files with their test atoms in impact data.

  • During selection, Smarter Testing can determine which test atom to run when a test file is modified.

The test atom to map can be specified in one of two ways:

  • Use the template variable << test.atoms >> in the file-mapper command.

  • If the template variable is not found in the file-mapper command, the test atom will be passed on stdin.

File mapper can run in two modes:

  • One test atom at a time. This mode expects a list of files back for a passed in << test.atom >>.

  • All test atoms in a single pass. This mode supports << outputs.go-list-json >> to map all go test files to go packages.

Key Required Type Description

file-mapper

N

String

A command that determines which test files contain the code for a test atom.

Example:

---
name: go tests
file-mapper: go list -json="Dir,ImportPath,TestGoFiles,XTestGoFiles" ./... > << outputs.go-list-json >>

analysis

The analysis command is similar to your run command, but with coverage instrumentation enabled. Analysis generates coverage data that tells Smarter Testing which source files were covered by each test atom.

Analysis can run in two modes:

  • One test atom at a time. This mode is compatible with most coverage tooling.

  • All test atoms in a single pass. This mode is much faster but requires specific support for the CircleCI coverage format from the coverage tooling.

The analysis command needs to be modified to use placeholders for the coverage report output location, and the test atom that should be run.

The test atom to analyze can be specified in one of two ways:

  • Use the template variable << test.atoms >> in the analysis command. This will be replaced with the test atom to analyze.

  • If the template variable is not found in the analysis command, the test atom will be passed on stdin.

Different template variables are available for coverage output, depending on the format of the coverage data produced by the test runner:

  • LCOV << outputs.lcov >> - analyses one test atom at a time.

  • Go’s coverage format << outputs.go-coverage >> - analyses one test atom at a time.

  • CircleCI coverage format << outputs.circleci-coverage >> - analyses all test atoms in a single pass.

Your analysis command should use the output variable for the coverage format it produces.

Key Required Type Description

analysis

N

String

A command that executes a test with coverage instrumentation.

Example:

---
name: go tests
analysis: go tool gotestsum -- -coverprofile="<< outputs.go-coverage >>" -cover -coverpkg ./... << test.atoms >>

outputs

A map of output fields that can be set to write command variables to known file paths.

When no output field is set and the variable is used in a command, a temporary file is created.

junit

The junit output is the path to write JUnit XML test results, used in the run command with << outputs.junit >> template variable.

Key Required Type Description

junit

N

String

File path to write JUnit XML test result reports.

Example:

# .circleci/config.yml
version: 2.1
jobs:
  test:
    executor: node-with-service
    steps:
      - setup
      - run: circleci run testsuite "ci tests"
      - store_test_results:
          path: test-reports
# .circleci/test-suites.yml
---
name: ci tests
outputs:
  junit: test-reports/junit.xml

options

A map of option fields that can be set to configure how the testsuite is run.

test-impact-analysis

To enable test impact analysis and only run test atoms relevant to your changes.

Key Required Type Description

test-impact-analysis

N

Boolean

Set true to enable test impact analysis.

Example:

# .circleci/test-suites.yml
---
name: ci tests
options:
  test-impact-analysis: true

test-analysis-duration

Set a target for the amount of time spent running analysis. Analysis may run slightly longer than the target in order to complete the current batch of work.

test-analysis-duration has no effect when running analysis in a single pass.

Key Required Type Description

test-analysis-duration

N

Integer (maximum 300 minutes)

The target number of minutes to spend running analysis.

Example:

# .circleci/test-suites.yml
---
name: ci tests
options:
  test-analysis-duration: 15

impact-key

Set a unique key for storing impact data.

Each independent testsuite should use a different impact key to avoid overwriting each other’s data.

There is no need to set this option if you only have a single test suite in .circleci/test-suites.yml.

Key Required Type Description

impact-key

N

String

Unique key to use for storing this testsuite’s impact data

Example:

# .circleci/test-suites.yml
---
name: ci tests
options:
  impact-key: production-ci-tests

dynamic-test-splitting

To enable dynamic test splitting and run test atoms from shared queue , reducing the impact of slow parallel nodes.

Key Required Type Description

dynamic-test-splitting

N

Boolean

Set true to enable dynamic test splitting.

Example:

# .circleci/test-suites.yml
---
name: ci tests
options:
  dynamic-test-splitting: true

full-test-run-paths

When test selection is enabled, changes to these file paths cause all test atoms to be selected and run.

Project files that affect the running system without directly referencing tests should be listed here. For example project dependency files, or database migration files.

The * and ** globs can be used to match files and directories.

Key Required Type Description

full-test-run-paths

N

List of strings

Example:

# .circleci/test-suites.yml
---
name: ci tests
options:
  full-test-run-paths:
    - go.mod
    - .circleci/*.yml
    - database-migrations/**/*.sql

test-selection-rules

Use test selection rules to extend the test selection mechanism to cover non-source files.

Each rule lists a test atom, and what should cause that test atom to be selected in addition to the usual selection algorithm.

The * and ** globs can be used to match files and directories.

Key Required Type Description

test-selection-rules

N

List of test-selection-rule

test-selection-rule

Key Required Type Description

test-atom

Y

String

Name of the test atom covered by this rule.

include

Y

Boolean, or list of file paths (limited to 100 globs/paths).

When to include the test atom in test selection. If true the test atom is always selected. If a list of file paths, then the test atom is selected when there are changes to any of those file paths.

Examples:

Run DB integration tests if any database migrations change, or new migrations are added.

# .circleci/test-suites.yml
---
name: ci tests
options:
  test-selection-rules:
    - test-atom: db/integration_test.ts
      include: database-migrations/**/*.sql

Always run acceptance tests.

# .circleci/test-suites.yml
---
name: ci tests
options:
  test-selection-rules:
    - test-atom: acceptance/test.ts
      include: true

max-auto-rerun

Set max-auto-rerun to automatically rerun failed test atoms up to a maximum number of attempts. Test atoms will run once initially, then up to a further max-auto-rerun times.

Key Required Type Description

max-auto-rerun

N

Integer (min 0, max 10)

The maximum number of times test atoms will be rerun if they fail.

When auto-rerun-duration is set, test atoms will rerun until one of the two conditions is reached.

Example:

# .circleci/test-suites.yml
---
name: ci tests
options:
  max-auto-rerun: 3

auto-rerun-duration

Set auto-rerun-duration to automatically rerun failed test atoms until either they all pass, or the rerun duration is reached.

Test atoms will rerun a maximum of 10 times even if the rerun duration is not reached.
Key Required Type Description

auto-rerun-duration

N

Duration or percentage

The duration or percentage of total runtime that will be spent rerunning test atoms if they fail, for example, 5m, 1h, 20%.

When max-auto-rerun is set, test atoms will rerun until one of the two conditions is reached.

Example:

# .circleci/test-suites.yml
---
name: ci tests
options:
  auto-rerun-duration: 5m