Automated testing in CircleCI

Cloud

Server v4+

Introduction

CircleCI allows you to automatically test your code before changes are merged. You can integrate testing tools and frameworks such as Jest, Mocha, pytest, JUnit, Selenium, XCTest, and more.

When you integrate your tests into your CircleCI pipelines, you not only deliver software to your users more reliably and efficiently, you get feedback more quickly so you can fix issues and failed tests faster. Test output data becomes available in CircleCI to help debug failed tests. If you save your test data in CircleCI, you can also take advantage of Test Insights as well as parallelism features to identify opportunities to further optimize your pipelines.

Basics

To automatically run your test suites in a project pipeline, you will add configuration keys in your .circleci/config.yml file. These would typically be defined as a step or collection of steps to be executed in a job.

A pipeline might consist of a single workflow, with a single job defined that includes a step to execute a suite of tests within an execution environment.

Using Docker? Authenticating Docker pulls from image registries is recommended when using the Docker execution environment. Authenticated pulls allow access to private Docker images, and may also grant higher rate limits, depending on your registry provider. For further information see Using Docker authenticated pulls.

jobs:
  build-and-test:
    docker:
      - image: cimg/node:16.10
    steps:
      - checkout
      - node/install-packages:
          pkg-manager: npm
      - run:
          name: Run tests
          command: npm test

run is a built-in step that runs commands in a shell. To read more about the run step for executing tests and other commands, go to the Configuration reference page.

Depending on your requirements, you might have more complex workflows that orchestrate multiple jobs. For example, you might have several concurrent jobs for building and testing your project in separate Linux, macOS, and Windows execution environments. You might also want to require that a test job is run only if a preceding build job is successful.

For a more in-depth look at workflows functionality, visit the Workflows page. You can also refer to the Sample configuration page for more workflow examples.

If you need to restrict the connections allowed in your test jobs, consider enabling IP ranges. This enables CircleCI jobs to go through a set of well-defined IP address ranges.

Use orbs to simplify testing

Orbs provide a way to integrate popular testing tools into your configuration. You can invoke CircleCI partner orbs such as Cypress, LambdaTest, and Sauce Labs in your .circleci/config.yml file. These orbs will allow you to include common testing tasks in your pipelines by running built-in jobs or concise usage commands in your jobs.

Orbs are available for mobile testing, browser testing, load testing, and code coverage. To get started with orbs, visit the Orbs introduction page. To view the orb registry, visit the CircleCI Developer Hub.

Store test data

Results from testing can be saved in CircleCI in two different ways.

Use the store_test_results step:
```
steps:
  ... # Steps to build and test your application
  - store_test_results:
      path: test-results
```
This step uploads and stores test results, and also enables direct access to output from failed tests on the Tests tab of a job in the web app.

More details on store_test_results can be found on the Configuration reference page.
Store test results as artifacts:

Test results can also be stored as artifacts using the store_artifacts step.
```
steps:
  ... # Steps to build and test your application
  - store_artifacts:
      path: test-results
      destination: junit
```
Results can later be accessed or downloaded as files via the Artifacts section of a job in the CircleCI web app, or through the API.

More details on store_artifacts can be found in the Configuration reference page. You can also read more in the Storing build artifacts guide.

For more detailed examples of storing test data with different testing frameworks, refer to the Collect test data page.

Timing tab

When parallelism and test splitting are enabled, CircleCI provides a bird’s-eye view into the timing of each parallel run in the Timing tab in the Job Details UI.

The Timing tab lets you identify which steps are taking the longest amount of time in a given parallel run so that you can make targeted improvements to reduce overall runtime.

Test insights

This feature is supported for orgs that use OAuth authentication.

Check your Organization slug at Organization settings Overview. OAuth authenticated orgs are structured as follows:

github/<your-org-name> or gh/<your-org-name>
bitbucket/<your-org-name> or bb/<your-org-name>

This feature is not supported if your organization slug is in the following format:

circleci/<UID>.

For an overview of feature support, see the VCS integration overview page.

When test results are stored, test analytics also become available on the Tests tab of the Insights page in the web app. Metrics for flaky tests, tests with the lowest success rates, and slow tests help you identify opportunities to optimize pipelines as well as further improve your testing strategy.

More information is available on the Test Insights page.

Re-run failed tests only (`circleci tests run`)

You can configure jobs to re-run failed tests only. Using this option, when a transient test failure arises, only a subset of tests are re-run instead of the entire test suite. Also, only failed tests from the same commit are re-run, not new ones.

More information on how to use this option is available on the Rerun failed tests overview page. This functionality uses a command called circleci tests run.

Historically, when your testing job in a workflow has flaky tests, the only option to get to a successful workflow was to re-run your workflow from failed. This type of re-run executes all tests from your testing job, including tests that passed, which prolongs time-to-feedback and consumes credits unnecessarily.

Next steps

Further optimize your pipelines with parallelism and test splitting.
Try our test splitting tutorial.
Integrate tests for macOS or iOS apps.
Read our Browser testing guide to common methods for running and debugging browser tests in CircleCI.
To get event-based notifications in Slack about your pipelines (for example, if a job passes or fails), try our Slack orb tutorial.