Documentation structure for LLMs (llms.txt)
CircleCI Logo

Getting started with CircleCI’s testing tool

1 day ago · 8 min read
Cloud

CircleCI’s testing tool, circleci testsuite, discovers and runs your tests. The following features are built-in and free:

Smarter Testing adds three features on top of circleci testsuite. You can enable each one independently:

  • Test impact analysis: run only the tests impacted by your code changes.

  • Dynamic test splitting: evenly distribute tests across parallel execution nodes as they run.

  • Auto rerun failed tests: automatically retry failed tests during the same run.

This guide walks you through the foundational setup that the built-in features and Smarter Testing features build on:

  1. Configure a .circleci/test-suites.yml file to discover and run your tests.

  2. Validate and run your tests using the testsuite command.

Before you begin

You need a repository with an existing test suite. The following setup process guides you through configuring JUnit XML output and running tests.

You can complete local steps (setup and run locally) without a CircleCI account. A CircleCI account with a project connected to your VCS is only required when you run tests in CI.

You can use the CircleCI Agent Skills to get set up for CircleCI’s testing tool. Install the CircleCI Public Skills plugin to your AI coding agent and ask your agent to get set up for CircleCI’s testing tool.

Terminology

The following terms are used throughout the CircleCI testing tool documentation:

Test atom

The smallest unit of work your test runner can execute independently - typically a test file (for example, tests/auth_test.py), but can also be a test package or module.

Test suite

A named configuration in .circleci/test-suites.yml that defines how to discover and run your test atoms.

1. Setup

Use the doctor command to set up and validate your test-suites.yml. The CLI runs through a series of checks, executing tests and validating the output. If any results look incorrect, action items are provided to resolve them.

  • Local

  • CI

Ensure the latest CircleCI CLI is installed to validate locally.

Run the doctor command in your terminal where you usually run tests.

$ circleci testsuite doctor "ci tests"

The CLI is already available in CI, no install is needed.

Run the doctor command in your job with your usual test setup.

version: 2.1
jobs:
  doctor:
    executor: docker
    steps:
      - setup
      - run: circleci testsuite doctor "ci tests"

Follow the steps until all checks pass.

Run the circleci testsuite CLI tooling from the directory where your tests are located. That directory is typically your repository root, but for monorepos it can be the root of a subpackage (for example, cd service-1 && circleci testsuite run "ci tests").

The testsuite will look for the .circleci/test-suites.yml configuration file in the repository root or the subpackage where the CLI is run. All commands (discover, run, analysis) execute relative to where you run the CLI, so avoid using cd or --directory flags within your commands.

2. Run

  • Local

  • CI

Run the test command in your terminal where you usually run tests.

$ circleci testsuite run "ci tests"

In your CircleCI configuration file, replace your existing test command with circleci testsuite run "ci tests" — the same command you used to setup. Keep all other job steps the same.

Verify that store_test_results points to the directory that matches outputs.junit in your test-suites.yml file.

version: 2.1
jobs:
  test:
    executor: node-with-service
    steps:
      - setup
      # Replace previous tests command e.g. vitest run --reporter=junit ...
      - run: circleci testsuite run "ci tests"
      - store_test_results:
          # This directory must match the directory of `outputs.junit` in your
          # test-suites.yml
          path: test-reports

Commit both .circleci/test-suites.yml and .circleci/config.yml to your feature branch and push to your VCS.

3. Verify in CI

After your pipeline runs, verify the test job behaves as expected before merging:

  1. In the CircleCI web app, navigate to your pipeline and open the test job.

  2. Check the Tests tab and confirm the number of test results matches what you see when running tests without the testsuite command. A difference in test results could indicate a misconfiguration in your discover command.

  3. Compare the job duration to a recent run without the testsuite command. The runtime is expected to be similar. A significant difference may indicate a misconfiguration in your test-suites.yml.

Rerun failed tests

Rerunning only failed tests is built in to circleci testsuite run and is free, with no extra configuration. When you rerun a workflow from failed, circleci testsuite run reruns only the tests that failed in the previous run, instead of your entire test suite. You can rerun a workflow from failed in any of the following ways:

To rerun failed tests automatically during the same run, use the Smarter Testing feature Auto Rerun Failed Tests.

Test splitting

Static test splitting is built in to circleci testsuite run and is free. circleci testsuite run splits your tests across parallel nodes when parallelism is set on your job, using the durations reported in your JUnit results.

To keep parallel nodes evenly loaded as they run, use the Smarter Testing feature Dynamic Test Splitting.

Next steps

All of your tests now run locally and in CI using the testsuite command. Next, enable the Smarter Testing features that fit your needs. Each feature can be enabled independently:

For the full set of configuration keys available in test-suites.yml, refer to the Test Suite Config Reference.