Documentation structure for LLMs (llms.txt)
CircleCI Logo

Auto rerun failed tests Beta

25 days ago · 6 min read
Cloud

Smarter Testing is available in beta. This means the product is in early stages and you may encounter bugs, unexpected behavior, or incomplete features. When the feature is made generally available, there will be a cost associated with access and usage.

Refer our Discuss post for more information about our beta launch.

Auto rerun failed tests automatically detects and retries test failures during your test run. Compared with the existing Rerun Failed Tests feature that CircleCI provides, Auto rerun failed tests eliminates the need to:

  • Manually identify which jobs failed.

  • Manually select the Rerun failed tests option in the CircleCI web app.

Is my project a good fit for auto rerun failed tests?

Auto rerun failed tests reruns tests within the same step automatically until all tests pass or the maximum rerun attempts is reached.

If your test suite has flaky tests, auto rerun can quickly recover from test failures without rerunning your entire test suite.

How it works

All tests are run as normal with the discover and run command.

When auto rerun failed tests is enabled:

  • Any test results that failed in the JUnit XML output are rerun.

  • The process repeats until all tests are successful, or the maximum number of rerun attempts is reached.

  • If all tests are successful, the errors are suppressed and the job succeeds.

  • If the maximum number of reruns is reached and there are still test failures, the job fails.

flowchart LR subgraph Run direction TB R1a[A pass] --- R1b[B fail] --- R1c[C pass] --- R1d[D fail] --- R1e[E fail] end subgraph Rerun-1[Rerun 1] direction TB R2b[B pass] --- R2d[D fail] --- R2e[E pass] end subgraph Rerun-2[Rerun 2] direction TB R3d[D pass] end Run --> Rerun-1 --> Rerun-2 --> Done[All passed] style R1a fill:#00DB74,color:#fff style R1c fill:#00DB74,color:#fff style R1b fill:#FC5656,color:#fff style R1d fill:#FC5656,color:#fff style R1e fill:#FC5656,color:#fff style R2b fill:#00DB74,color:#fff style R2e fill:#00DB74,color:#fff style R2d fill:#FC5656,color:#fff style R3d fill:#00DB74,color:#fff style Done fill:#00DB74,color:#fff linkStyle 0 stroke:none linkStyle 1 stroke:none linkStyle 2 stroke:none linkStyle 3 stroke:none linkStyle 4 stroke:none linkStyle 5 stroke:none

Prerequisites

Before enabling auto rerun failed tests, ensure you have completed the Getting Started With Smarter Testing guide and have:

  • Installed the testsuite CLI plugin.

  • Configured your .circleci/test-suites.yml with discover and run commands.

  • Verified your tests run successfully with the testsuite command.

  • Correctly configured JUnit XML output in .circleci/test-suites.yml.

1. Enable auto rerun failed tests option in your .circleci/test-suites.yml file

---
name: ci tests
# ...
options:
  max-auto-rerun: 3
Options Field Default Description

max-auto-rerun

0

The maximum number of times tests will be rerun if they fail. max-auto-rerun can be set in the range 0 to 10 inclusive.

When auto-rerun-duration is set, tests will rerun until one of the two conditions is reached.

auto-rerun-duration

null

The duration or percentage of total runtime tests will be rerun if they fail, for example, 5m, 1h, 20%.

When max-auto-rerun is set, tests will rerun until one of the two conditions is reached.

2. Run locally

Use --doctor locally to validate the test-suites.yml is set up correctly.

The CLI will run through additional checks when auto rerun is enabled, if any results look incorrect an action item is provided to resolve it. Follow the steps until all checks pass.

$ circleci run testsuite "ci tests" --doctor

The circleci run testsuite CLI tooling should be run from the directory where your tests are located. This is typically your repository root, but for monorepos it can be the root of a subpackage (for example, cd service-1 && circleci run testsuite "ci tests").

The testsuite will automatically find the .circleci/test-suites.yml configuration file by walking up the directory tree. All commands (discover, run, analysis) execute relative to where you run the CLI, so avoid using cd or --directory flags within your commands.

Modify a test to cause it to fail and check that the test is rerun locally by removing the --doctor flag.

$ circleci run testsuite "ci tests"

3. Run in CI

Commit .circleci/test-suites.yml to your feature branch and push to your VCS.

Verify auto rerun failed tests is working as expected in CI by looking at the "Test" tab in the UI.

Next steps