CircleCI Logo
Navigation Menu Icon

Automatic reruns

Language Icon 14 days ago · 11 min read
Cloud
Contribute Go to Code

Automatic reruns help reduce the impact of temporary failures in your CI/CD pipelines. When a workflow fails due to transient issues, CircleCI can automatically restart the failed step or workflow without manual intervention.

Common use cases include:

  • Handling flaky tests.

  • Managing unreliable infrastructure or network connectivity problems.

  • Working with spot instances that may be terminated unexpectedly.

  • Preventing merge queue delays caused by transient failures.

Introduction

Automatic reruns provide a safety net for your CI/CD pipelines by automatically retrying failed steps and/or workflows. Automatic reruns help teams maintain productivity by reducing the need for manual intervention when steps and workflows fail due to temporary issues.

When you configure automatic reruns, CircleCI monitors the status of your pipelines and reruns work as required.

Benefits of automatic reruns include:

  • Reduced manual intervention for transient failures.

  • Improved pipeline reliability and developer productivity.

  • Cost-effective retry strategy that only reruns failed jobs.

  • Configurable retry limits to prevent infinite loops.

Quickstart

This section provides config examples to help you set up automatic reruns. The content following this quickstart section details how the feature works, along with some troubleshooting guidance and frequently asked questions.

Automatic step reruns

To enable automatic reruns for a run step in a job, add the max_auto_reruns key to your step with a value between 1 and 5:

Automatic reruns are only supported for run steps, not special steps like checkout or setup_remote_docker. Also you must configure the command key for the step, you cannot use the short form run step configuration.
CircleCI config to automatically retry a step up to 3 times if it fails
version: 2.1

jobs:
  my-job:
    steps:
      - run: echo "Hello, world!"
      - run:
          command: echo "This step will automatically rerun up to 3 times if it fails"
          max_auto_reruns: 3

You can add a delay between the step failing and the automatic rerun starting, as follows:

If you configure a delay for your automatic step rerun, the delay must be less than or equal to 10 minutes. The delay can be configured as a string with a unit suffix, using m (minutes) or s (seconds), for example, auto_rerun_delay: 10s or auto_rerun_delay: 3m. If you do not supply a delay, the rerun will start immediately after the step fails.
CircleCI config to automatically retry a step up to 3 times if it fails with a 10 second delay between attempts
version: 2.1

jobs:
  my-job:
    steps:
      - run: echo "Hello, world!"
      - run:
          command: echo "This step will automatically rerun up to 3 times if it fails with a 10 second delay between attempts"
          max_auto_reruns: 3
          auto_rerun_delay: 10s

Automatic workflow reruns

To enable automatic reruns for a workflow, add the max_auto_reruns key to your workflow with a value between 1 and 5:

CircleCI config to automatically retry my-workflow up to 3 times if it fails
version: 2.1

workflows:
  my-workflow:
    max_auto_reruns: 3
    jobs:
      - build
      - test
      - deploy:
          requires:
            - build
            - test

You can combine automatic reruns with workflow conditions. The following example shows how to configure automatic reruns for a workflow that is not triggered by a schedule trigger:

Combining automatic reruns with workflow conditions
version: 2.1

workflows:
  test-workflow:
    when:
      not:
        equal: [ scheduled_pipeline, << pipeline.trigger_source >> ]
    max_auto_reruns: 4
    jobs:
      - build
      - test

How automatic reruns work

Automatic step reruns rerun a full failed step.

Automatic workflow reruns function similarly to manually selecting Rerun workflow from failed in the CircleCI web app. When a workflow fails, CircleCI evaluates whether the workflow qualifies for automatic rerun based on specific criteria, as described in the following section.

When both automatic step reruns and automatic workflow reruns are configured together for the same job, the following rules apply:

  • The automatic step rerun triggers first.

  • If the workflow still fails after all step rerun attempts, the automatic workflow rerun triggers.

Automatic rerun criteria

CircleCI monitors workflow completion status and automatically triggers a "rerun from failed" operation when the following criteria are met. This ensures that only failed jobs and their dependencies are rerun, while successful jobs from the original workflow are not repeated.

  • The workflow status is "failed".

  • The max_auto_reruns value is specified in the configuration.

  • The number of remaining rerun attempts is greater than zero, and less than or equal to the specified max_auto_reruns value.

  • The workflow is not a manual rerun.

  • The pipeline is not older than 90 days.

CircleCI monitors step completion status and automatically reruns a failed step when the following criteria are met:

  • The step status is "failed".

  • The max_auto_reruns value is specified for the step in the configuration.

  • The number of remaining rerun attempts is greater than zero, and less than or equal to the specified max_auto_reruns value.

Automatic workflow rerun behavior

When an automatic workflow rerun is triggered:

  • Only failed jobs from the original workflow are retried. If the previous failure blocked dependent jobs from running, these jobs are also run.

  • Successfully completed jobs are not rerun.

  • The rerun uses the same actor permissions as the original workflow.

Monitoring automatic reruns

CircleCI provides several ways to monitor and track automatic rerun activity.

UI indicators

Automatic workflow reruns are indicated on the pipelines page in the CircleCI web app. In the Trigger event column you sill see Auto-rerun followed by the rerun attempt number, as shown in the following screenshot.

In this example the workflow is rerun twice out of a possible five attempts before it succeeds.

Automatic reruns UI
Figure 1. Automatic workflow reruns in the CircleCI web app

Automatic step reruns are indicated on the job page in the CircleCI web app.

In this example the run tests step is configured to rerun up to 3 times if it fails with a delay of one minute between attempts.

Automatic step reruns UI
Figure 2. Automatic step reruns in the CircleCI web app

Get details via the API

You can get information about automatic reruns via the CircleCI APIs.

Automatic workflow reruns

You can retrieve information about automatic workflow reruns using the CircleCI API v2:

curl -X GET "https://circleci.com/api/v2/workflow/{workflow-id}" \
  -H "Circle-Token: YOUR_TOKEN"

The API response includes the following fields for automatic workflow reruns:

  • auto_rerun_number: The current rerun attempt number.

  • max_auto_reruns: The maximum number of reruns configured.

Automatic step reruns

You can retrieve information about automatic step reruns using the job information API endpoint from the CircleCI API v1.1.

curl https://circleci.com/api/v1.1/project/:vcs-type/:username/:project/:build_num -H "Circle-Token: <circle-token>"

The API response includes the following fields for automatic step reruns:

  • auto_rerun_number: The current rerun attempt number.

  • max_auto_reruns: The maximum number of reruns configured.

Limitations

Be aware of these limitations when using automatic workflow reruns:

  • Maximum rerun attempts are capped at 5 per step and 5 per workflow.

  • Only the original workflow triggers automatic reruns. Manual reruns do not trigger automatic reruns.

  • Automatic reruns are disabled if the pipeline is older than 90 days.

  • Only failed workflows trigger automatic reruns, not cancelled workflows.

  • Automatic step reruns are only supported for run steps, not special steps like checkout or setup_remote_docker.

Troubleshooting

Common issues and solutions for automatic workflow reruns.

Reruns not triggering

If automatic reruns are not starting, check these conditions:

  • Verify max_auto_reruns is specified in your configuration.

  • Ensure the step or workflow status is "failed" and not "cancelled".

  • Confirm the maximum rerun attempts have not been exceeded.

  • Check that the workflow was not manually rerun.

  • Verify the pipeline is less than 90 days old.

# Correct configuration
workflows:
  my-workflow:
    max_auto_reruns: 3  # Must be present
    jobs:
      - build

Excessive rerun attempts

To prevent unnecessary reruns and credit consumption:

  • Set conservative max_auto_reruns values based on your failure patterns.

  • Investigate recurring failures to address root causes.

  • Monitor rerun patterns to optimize configuration.

Configuration errors

Common configuration mistakes include:

  • Setting max_auto_reruns greater than 5 (results in configuration error).

  • Placing max_auto_reruns at the job level instead of workflow level.

# Incorrect - job level
jobs:
  build:
    max_auto_reruns: 3  # Wrong placement
    docker:
      - image: cimg/base:2021.04

# Correct - workflow level
workflows:
  my-workflow:
    max_auto_reruns: 3  # Correct placement
    jobs:
      - build

Frequently asked questions

Do automatic reruns consume additional credits?

Yes, automatic reruns consume compute credits for each retry attempt. Only failed jobs are rerun, so successful jobs from the original workflow do not consume additional credits.

What happens if I manually rerun a workflow?

If you manually rerun a workflow and it fails, no automatic reruns will be triggered for the manually rerun workflow.

Do automatic reruns work with approval jobs?

Yes.

Do automatic reruns work with restricted contexts?

Yes, automatic reruns use the same actor permissions as the original workflow, so they work with restricted contexts as long as the original workflow had the necessary permissions.

Can I add a delay between automatic reruns?

Automatic reruns start immediately after the workflow fails.

Can I configure automatic reruns at the step level?

Yes, you can configure automatic reruns at the step or workflow level.