Documentation structure for LLMs (llms.txt)
CircleCI Logo

Automated testing in CircleCI

14 days ago
Cloud Server

CircleCI allows you to automatically test your code before changes are merged. You can integrate testing tools and frameworks such as Jest, Mocha, pytest, JUnit, Selenium, XCTest, and more. This page covers the basics that apply to every project, then helps you choose your test tooling.

Run tests in CI

When you integrate your tests into your CircleCI pipelines, you deliver software to your users more reliably and efficiently, and get faster feedback. This also means you can fix issues and failed tests faster. Test output data becomes available in CircleCI to help debug failed tests. If you save your test data in CircleCI, you can also split your tests across parallel execution nodes to reduce test time. Test Insights then helps you find flaky and slow tests.

To automatically run your test suites in a project pipeline, you will add configuration keys in your .circleci/config.yml file. These would typically be defined as a step or collection of steps to be executed in a job.

A pipeline might consist of a single workflow, with a single job defined that includes a step to execute a suite of tests within an execution environment.

Using Docker? Authenticating Docker pulls from image registries is recommended when using the Docker execution environment. Authenticated pulls allow access to private Docker images, and may also grant higher rate limits, depending on your registry provider. For further information, see Using Docker Authenticated Pulls.
Run tests directly in CI
jobs:
  build-and-test:
    docker:
      - image: cimg/node:24.16.0
    steps:
      - checkout
      - node/install-packages:
          pkg-manager: npm
      - run:
          name: Run tests
          command: npm test

run is a built-in step that runs commands in a shell. To read more about the run step for executing tests and other commands, go to the Configuration Reference page.

This example runs your test command directly. On CircleCI Cloud, the recommended way to run tests is to replace that command with circleci testsuite run, which adds Smarter Testing capabilities such as test splitting and rerunning only failed tests. See Choose your test tooling.

For example, for a project that uses Vitest, the job runs circleci testsuite run in place of the test command:

.circleci/config.yml using Smarter Testing
jobs:
  build-and-test:
    docker:
      - image: cimg/node:24.16.0
    steps:
      - checkout
      - node/install-packages:
          pkg-manager: npm
      - run:
          name: Run tests
          command: circleci testsuite run "ci tests"
      - store_test_results:
          # Must match outputs.junit in test-suites.yml
          path: test-reports

A .circleci/test-suites.yml file tells circleci testsuite how to discover and run your tests:

.circleci/test-suites.yml
name: ci tests
discover: npx vitest list --filesOnly
run: npx vitest run --reporter=junit --outputFile="<< outputs.junit >>" --silent --bail 0 << test.atoms >>
outputs:
  junit: test-reports/junit.xml

Depending on your requirements, you might have more complex workflows that orchestrate multiple jobs. For example, you might have several concurrent jobs for building and testing your project in separate Linux and Windows execution environments. You might also want to require that a test job is run only if a preceding build job is successful.

For a more in-depth look at workflows functionality, visit the Workflows page. You can also refer to the Sample configuration page for more workflow examples.

If you need to restrict the connections allowed in your test jobs, consider enabling IP Ranges. This enables CircleCI jobs to go through a set of well-defined IP address ranges.

Timing tab

The Timing tab in the Job Details UI shows how long each step of a job took. Use it to identify which steps take the most time, so that you can make targeted improvements to reduce overall runtime.

When a job runs with parallelism and test splitting, the Timing tab shows the timing of each parallel run.

Timing tab
Figure 1. Timing tab, parallel runs

Test uploads

Use the store_test_results step to upload your test results to CircleCI:

steps:
  ... # Steps to build and test your application
  - store_test_results:
      path: test-results

This step uploads and stores test results, and also enables direct access to output from failed tests on the Tests tab of a job in the web app.

More details on store_test_results can be found on the Configuration Reference page. For more detailed examples of storing test data with different testing frameworks, refer to the Upload Test Results page.

Test insights

This feature is supported for orgs that use OAuth authentication.

Check your Organization slug at Organization settings  Overview. OAuth authenticated orgs are structured as follows:

  • github/<your-org-name> or gh/<your-org-name>

  • bitbucket/<your-org-name> or bb/<your-org-name>

This feature is not supported if your organization slug is in the following format:

  • circleci/<UID>

For an overview of feature support, see the VCS Integration Overview page.

When test results are stored, test analytics also become available on the Tests tab of the Insights page in the web app. Metrics for flaky tests, tests with the lowest success rates, and slow tests help you identify opportunities to optimize pipelines as well as further improve your testing strategy.

More information is available on the Test Insights page.

Store test results as artifacts

You can also store test results as artifacts using the store_artifacts step. Artifacts are files you can download later. They do not appear on the Tests tab.

steps:
  ... # Steps to build and test your application
  - store_artifacts:
      path: test-results
      destination: junit

Results can later be accessed or downloaded as files via the Artifacts section of a job in the CircleCI web app, or through the API.

More details on store_artifacts can be found in the Configuration Reference page. You can also read more in the Storing Build Artifacts guide.

Choose your test tooling

On CircleCI Cloud, use Smarter Testing to run your tests. Smarter Testing replaces your test command with circleci testsuite run, which discovers and runs your tests. You get the following benefits:

  • Free capabilities with no extra configuration: static test splitting across parallel nodes, rerunning only failed tests when you rerun a workflow from failed, and test uploads.

  • Faster test runs when you need them: test impact analysis runs only the tests impacted by your code changes. Dynamic test splitting keeps parallel nodes evenly loaded. Dynamic parallelism sets the number of parallel nodes when the pipeline runs. Auto rerun failed tests retries flaky tests automatically. These capabilities have a credit charge above your plan’s allowance.

  • The same command locally and in CI: run and validate your test setup on your own machine before you push.

To learn more, read the Smarter Testing Overview, then follow Getting Started With Smarter Testing.

You can use the CircleCI Agent Skills to get set up for Smarter Testing. Install the CircleCI Public Skills plugin to your AI coding agent and ask your agent to get set up for Smarter Testing.

If your project already uses the circleci tests glob, circleci tests split, or circleci tests run commands, they continue to work. To move them to Smarter Testing, follow Getting Started With Smarter Testing.

Next steps