CircleCI Logo
Navigation Menu Icon

Auto rerun failed tests Preview

Language Icon 9 days ago · 4 min read
Cloud
Contribute Go to Code
Smarter Testing is available in preview. This means the product is in early stages and you may encounter bugs, unexpected behavior, or incomplete features. When the feature is made generally available, there will be a cost associated with access and usage.

Auto rerun failed tests automatically detects and retries test failures during your test run. This allows you to quickly recover from flaky test failures without rerunning your entire test suite. Compared with the existing Rerun Failed Tests feature that CircleCI provides, Auto rerun failed tests eliminates the need to:

  • Manually identify which jobs failed.

  • Manually select the Rerun failed tests option in the CircleCI web app.

How it works

All tests are run as normal with the discover and run command.

When auto rerun failed tests is enabled:

  • Any test results that failed in the JUnit XML output are rerun.

  • The process repeats until all tests are successful, or the maximum number of rerun attempts is reached.

  • If all tests are successful, the errors are suppressed and the job succeeds.

  • If the maximum number of reruns is reached and there are still test failures, the job fails.

Prerequisites

Before enabling auto rerun failed tests, ensure you have completed the Getting Started With Smarter Testing guide and have:

  • Installed the testsuite CLI plugin.

  • Configured your .circleci/test-suites.yml with discover and run commands.

  • Verified your tests run successfully with the testsuite command.

  • Correctly configured JUnit XML output in .circleci/test-suites.yml.

Configure JUnit XML output

To determine which test have failed, the test atoms from the discover command must match one the JUnit XML output fields:

  • testname

  • classname

  • filename

For example, if the discover command outputs the test atom src/feature.test.ts, this value must exist in one of the JUnit XML fields listed above (from the JUnit output generated by the run command) to determine if it failed. If a test atom cannot be matched with a result, the following warning is output and the test is not rerun.

failed to match test atom with test result

See Configure the Run Command in the Getting Started with Smarter Testing guide to set up JUnit XML output.

Auto rerun failed tests configuration options

The following options are available to be defined in the options map in your .circleci/test-suites.yml configuration file:

Options Field Default Description

max-auto-rerun

0

The maximum number of times tests will be rerun if they fail. max-auto-rerun can be set in the range 0 to 10 inclusive.

When auto-rerun-duration is set, tests will rerun until one of the two conditions.

auto-rerun-duration

null

The duration or percentage of total runtime tests will be rerun if they fail, for example, 5m, 1h, 20%.

When max-auto-rerun is set, tests will rerun until one of the two conditions.

After you set max-auto-rerun to a value greater than 0, you have successfully set up auto rerun failed tests. Next, you can enable other features of Smarter Testing: