Go to Application

Set up a rollback pipeline

1 day ago

Cloud

Contribute

CircleCI rollbacks allow you to quickly revert your application to a previous stable version when issues are detected in production. This guide covers how to create and configure a CircleCI pipeline to enable rollback operations.

Introduction

CircleCI rollbacks support two methods of manually reverting to a previous version:

Rollback by running a custom rollback pipeline.
Rollback by rerunning the workflow that originally deployed the version you want to revert to.

This guide covers setting up a rollback pipeline, which is the recommended approach for most applications. The process of rolling back is described in detail in the Rollback a Deployment guide, including a comparison of the two methods.

Figure 1. Rollback guided setup

Prerequisites

A CircleCI account connected to your code. You can sign up for free.
Your code must be stored in a GitHub repository to use the rollback pipeline feature. You will need to install the CircleCI GitHub App into your organization if you have not already done so.
A CircleCI project with a workflow configured to deploy your code.
Deploy markers must be configured in your project to use CircleCI rollbacks. You will be guided to configure deploy markers through the steps on this page or you can get set up manually by following the Configure Deploy Markers guide.
Familiarity with CircleCI deploys concepts. Find full details in the CircleCI Deployment Overview.

Set up a rollback pipeline

The following steps guide you through setting up a custom rollback pipeline. Using a custom rollback pipeline is the recommended rollback method.

1. Start the rollback pipeline setup

In the CircleCI dashboard, navigate to Home or Projects from the sidebar.
Select the Overview link for your project.
Select Rollback and from the dropdown select Create rollback pipeline.

Figure 2. Rollback button on project overview page

2. Confirm GitHub App installation

To set up rollback pipelines you must install the CircleCI GitHub App into your organization. For more information, see the Users, Organizations, and Integrations Guide.

Only organization administrators can install the GitHub App. If you are not an organization administrator, you can ask an administrator to install the GitHub App for you.

If the GitHub App is not installed in your organization, install it now by selecting Install the GitHub App. If the GitHub App is already installed in your organization, the GitHub App Installation step is automatically marked as completed.

Figure 3. Install the GitHub App

3. Create a pipeline definition

Next, create a new pipeline definition for your rollback pipeline.

A pipeline definition is a setting within CircleCI in your project to specify the different elements that make up a pipeline:

Config source
Config file path
Checkout source
Triggers

No changes are pushed to your repository at this point. GitHub pipelines are viewable at Project settings Project Setup.

Select the repository that includes your project code. This is the repository that you want to rollback with the pipeline you are setting up.
Select Create pipeline definition. This creates a new pipeline definition for your rollback pipeline in your project.

4. Add deploy markers

Deploy markers must be configured in your project to use CircleCI rollback pipelines. If you already have deploy markers set up, select "I’ll make the config changes myself" and then select I’ve updated my config to proceed to the next step.

The in-app setup process guides you to set up deploy markers now. Alternatively, manual configuration steps are available in the Configure Deploy Markers guide.

Generate with AI
Make the changes yourself

Select Generate config change with AI to auto-generate a new configuration file, based on your project configuration, with deploy markers added.
Once the generation is complete, select Create pull request to create a pull request with the changes.
View and check the pull request and merge when you are satisfied with the changes.
Back in the CircleCI web app, select I have merged the pull request to proceed to the next step.

Select the "I’ll make the config changes myself" checkbox.
In the Environment Name field, select or type your target environment name. More information in the Configure Deploy Markers guide. If the specified environment does not exist, it will be created. If you do not specify an environment, CircleCI will create one named default.
In the Component Name field, select or type the name of the component you want to rollback. The component name sets the name that will be displayed in the Deploys UI.

The setup will auto-generate deploy marker commands based on your selections.

Figure 4. Add deploy marker values
Copy the generated commands using the Copy button and paste them into your existing config file in your repository where the deployment occurs on the branch you selected in the guided setup.

Ensure the circleci run release plan step comes before the deployment step, followed by the circleci run release update step.

You will need to set the target version for these commands. See the Examples for Target Version section for more details.

Figure 5. Copy deploy marker commands to your config file
Once you have updated your config file, select I’ve updated my config to proceed to the next step.

5. Set up rollback config

In this step you will configure your new rollback pipeline. You can then use this to revert to a stable version of your application from the CircleCI web app.

Review the example config file template. It shows how you will perform rollbacks and how to use deploy markers to track these rollbacks
Select Commit config & create Pull Request to create this rollback configuration template in your repository. The setup will open a pull request in a new tab with the suggested changes.

The configuration will be committed to a new branch called rollback-pipeline-setup in the repository you selected earlier in this guided setup.

Figure 6. Commit config for rollback pipeline and open a PR

Alternatively, you can copy the suggested configuration into an existing or newly created rollback config file. Call the file rollback.yml and save it in your repository’s .circleci folder, for the system to detect and use this file. If you choose this method, select I already have a rollback config and skip to the next step. See the Configuration tips section for more details on customizing your rollback configuration.
Navigate to the pull request in your repository by selecting View pull Request under the example config file.
In the pull request, modify the rollback configuration according to your specific deployment needs. For example, set your own parameters and rollback logic, uncomment the included common rollback setups or write your own custom rollback implementation.
Once you have customized the configuration, merge the pull request to complete the rollback setup. Head back to the CircleCI web app and select Setup rollback pipeline to activate the rollback pipeline.

You can now rollback deployments by running your rollback pipeline. More information is available in the Rollback a Deployment guide.

Run a rollback pipeline

For steps to running a rollback pipeline, see the Rollback a Deployment guide.

Change rollback pipeline

You can switch rollback pipelines for your project at any time. If you have configured a new pipeline and want to trigger this pipeline when performing rollbacks, you can change the rollback pipeline as follows:

In the CircleCI web app, select your org from the org cards on your user homepage.
Select Projects from the sidebar and locate your project from the list. You can use the search to help.
Select the ellipsis next to your project and select Project Settings.

You can also access project settings from each project overview page using the Settings button.
Select the Deploys tab.
In the Rollback Pipeline section, choose the pipeline you want to be selected as the rollback pipeline from the dropdown. You can also choose the unset the rollback pipeline at this point, which will enable you to use the guided setup again as described above.

Figure 7. Change rollback pipeline

Configuration tips

When customizing your rollback configuration, you can use the following pipeline values to access rollback values:

pipeline.deploy.component_name
pipeline.deploy.environment_name
pipeline.deploy.target_version
pipeline.deploy.current_version
pipeline.deploy.namespace
pipeline.deploy.reason

For a full list of pipeline values, see the Pipeline Values guide.

Example rollback pipeline configuration

In this section you can find a full example of a rollback pipeline config. This example uses Helm to perform a rollback on AWS EKS and kubectl to validate its status.

This template assumes the following:

Your deployment is annotated with a "version" label.
The name of your Helm chart is the same as the name of the component. If this is not the case, you can instead add a label to the deployment with the component name and retrieve it that way

Example rollback pipeline configuration

version: 2.1

orbs:
  aws-cli: circleci/aws-cli@5.4.0
  helm: circleci/helm@3.2.0

commands:
  # The following command is needed only for the specific logic in this example. Feel free to remove it if you don't need it.
  install_yq:
    steps:
      - run:
          name: install yq
          command: |
            wget https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 -O /tmp/yq
            wget https://github.com/mikefarah/yq/releases/latest/download/checksums
            sha_file=$(sha256sum /tmp/yq | awk '{ print $1 }')
            sha=$(awk '$1=="yq_linux_amd64"{print $19}' checksums)
            if [ "$sha_file" != "$sha" ]; then
                    echo "Checksum failed" >&2
                    exit 1
            fi
            echo "The checksums match."
            chmod +x /tmp/yq
  verify_current_version:
    description: "Verifies that the currently deployed version matches the expected value"
    parameters:
      resource_name:
        type: string
        description: "Name of the resource to roll back"
      namespace:
        type: string
        default: "default"
        description: "Kubernetes namespace (optional)"
      current_version:
        type: string
        description: "Current version"
    steps:
      - run:
          name: Verify current version
          command: |
            RELEASE_NAME="<< parameters.resource_name >>"
            if [ "<< parameters.current_version >>" == "" ]; then
              echo "Current version not specified."
              exit 0
            fi
            if [ -z "$RELEASE_NAME" ]; then
              echo "Missing release name"
              exit 1
            fi
            helm get manifest "<< parameters.resource_name >>" --namespace "<< parameters.namespace >>" > manifest.yaml
            VERSION_LABEL=$(yq e '
              select(.kind == "Deployment") |
              .spec.template.metadata.labels.version
            ' manifest.yaml)
            if [ -z "$VERSION_LABEL" ] || [ "$VERSION_LABEL" == "null" ]; then
              echo "Could not extract version label from manifest"
              exit 1
            fi
            if [ "$VERSION_LABEL" == "<< parameters.current_version >>" ]; then
              echo "Version matches input version << parameters.current_version >>"
            else
              echo "Version mismatch: expected << parameters.current_version >> but found $VERSION_LABEL"
              exit 1
            fi
  retrieve_target_revision:
    description: "Retrieve previous version"
    parameters:
      resource_name:
        type: string
        description: "Name of the resource to roll back"
      namespace:
        type: string
        default: "default"
        description: "Kubernetes namespace (optional)"
      target_version:
        type: string
        description: "Target version"
    steps:
      - run:
          name: Identify previous revision
          command: |
            TARGET_VERSION="<< parameters.target_version >>"
            RELEASE_NAME="<< parameters.resource_name >>"
            NAMESPACE="<< parameters.namespace >>"
            if [ -z "$TARGET_VERSION" ]; then
              echo "TARGET_VERSION is required"
              exit 1
            fi
            # Get full release history
            REVISIONS=$(helm history "$RELEASE_NAME" --namespace "$NAMESPACE" --output json | jq '.[].revision')
            if [ -z "$REVISIONS" ]; then
              echo "Could not fetch Helm history for release '$RELEASE_NAME'"
              exit 1
            fi
            # Search each revision for a Deployment with the matching version label
            TARGET_REVISION=""
            for REV in $REVISIONS; do
              helm get manifest "$RELEASE_NAME" --namespace "$NAMESPACE" --revision "$REV" > manifest.yaml || continue
              VERSION_LABEL=$(yq e '
                select(.kind == "Deployment") |
                .spec.template.metadata.labels.version
              ' manifest.yaml)
              if [ "$VERSION_LABEL" == "$TARGET_VERSION" ]; then
                TARGET_REVISION=$REV
                break
              fi
            done
            if [ -n "$TARGET_REVISION" ]; then
              echo "export CONTAINER_VERSION=${TARGET_VERSION}" >> $BASH_ENV
              echo "export TARGET_REVISION=${TARGET_REVISION}" >> $BASH_ENV
              source $BASH_ENV
            else
              echo "No revision found with version label: $TARGET_VERSION"
              exit 1
            fi
  perform_rollback:
    description: "perform rollback"
    parameters:
      resource_name:
        type: string
        description: "Name of the resource to roll back"
      namespace:
        type: string
        default: "default"
        description: "Kubernetes namespace (optional)"
    steps:
      - run:
          name: Perform rollback
          command: |
            helm rollback << parameters.resource_name >> ${TARGET_REVISION}
  # This command validates the deployment after rolling back. The provided example uses kubectl to check the ready replicas and number of restarts
  # of pods associated with the deployment and causes the job to fail if the deployment is not ready or has too many restarts by
  # the end of the validation duration.
  # Mind the fact that the example assumes you have an app label with value equal to the component name, in order to retrieve the pods.
  # If that is not the case you will have to adapt the logic in the script.
  validate_deployment:
    description: "Validates the deployment after rolling back"
    parameters:
      resource_name:
        type: string
        description: "Name of the resource that has been rolled back"
      namespace:
        type: string
        default: "default"
        description: "Kubernetes namespace (optional)"
      target_version:
        type: string
        description: "Target version"
      max_restarts:
        type: integer
        default: 5
        description: "Maximum number of allowed restarts"
      duration:
        type: integer
        default: 600
        description: "Duration of the validation in seconds"
    steps:
      - run:
          name: Validate deployment
          command: |
            CHECK_DURATION=$((SECONDS+<< parameters.duration >>))  # 10 minutes duration
            REPLICAS_OK=false

            LABEL_SELECTOR="app=<< parameters.resource_name >>,version=<< parameters.target_version >>"
            DEPLOYMENT_FOUND=false
            echo "Starting validation of version: << parameters.target_version >>"
            while [ $SECONDS -lt $CHECK_DURATION ]; do
              DEPLOYMENT=$(kubectl get deployment << parameters.resource_name >>  -n << parameters.namespace >> --ignore-not-found -o json)

              if [ -n "$DEPLOYMENT" ]; then
                  DEPLOYMENT_FOUND=true
                  DESIRED=$(echo "$DEPLOYMENT" | jq -r '.spec.replicas // 0')
                  READY=$(echo "$DEPLOYMENT" | jq -r '.status.readyReplicas // 0')

                  # Handle empty values
                  DESIRED=${DESIRED:-0}
                  READY=${READY:-0}

                  echo "Current replicas $READY/$DESIRED"
                  if [ "$DESIRED" -eq "$READY" ]; then
                    REPLICAS_OK=true
                  else
                    REPLICAS_OK=false
                  fi
              else
                  DEPLOYMENT_FOUND=false
                  echo "Deployment not found"
                  continue
              fi
              RESTARTS=$(kubectl get pods -l $LABEL_SELECTOR -n << parameters.namespace >> \
                -o jsonpath='{.items[*].status.containerStatuses[*].restartCount}' 2>/dev/null | awk '{sum=0; for(i=1; i<=NF; i++) sum+=$i; print sum+0}')
              # Handle potential errors
              if [[ -z "$RESTARTS" || ! "$RESTARTS" =~ ^[0-9]+$ ]]; then
                RESTARTS=0
              fi
              echo "Number of restarts $RESTARTS"
              if [ $RESTARTS -gt << parameters.max_restarts >> ]; then
                echo "FAILURE_REASON='Exceeded maximum number of restarts'" > failure_reason.env
                exit 1
              fi
              sleep 10  # Check every 10 seconds
            done
            if [ $DEPLOYMENT_FOUND = false ]; then
                echo "FAILURE_REASON='Deployment was not found'" > failure_reason.env
                exit 1
            fi
            if [ $REPLICAS_OK = false ]; then
                echo "FAILURE_REASON='Desired replicas doesn't match ready replica'" > failure_reason.env
                exit 1
            fi

jobs:
  rollback-component:
    docker:
      - image: cimg/aws:2023.03
    environment:
      COMPONENT_NAME: << pipeline.deploy.component_name >>
      NAMESPACE: << pipeline.deploy.namespace >>
      ENVIRONMENT_NAME: << pipeline.deploy.environment_name >>
      TARGET_VERSION: << pipeline.deploy.target_version >>
    steps:
      - checkout
      - attach_workspace:
          at: .
      ### Uncomment this section if you are using AWS EKS, otherwise add the steps to authenticate with your platform
      - aws-cli/setup:
          role_arn: $AWS_OIDC_ROLE
          region: $AWS_REGION
          role_session_name: "example"
          session_duration: "1800"
      - run: aws sts get-caller-identity
      - run: aws configure list
      - run:
          name: Update kubeconfig for EKS
          command: |
            aws eks update-kubeconfig --name "$EKS_CLUSTER_NAME"
            aws sts get-caller-identity  # Verify credentials are still valid
      - helm/install_helm_client
      - install_yq
      # This command is used to validate that the current version on your cluster matches the value that was specified when
      # the pipeline was triggered. If that is not the case it is possible that the deployment has been updated in the meantime
      # this check is optional and can be removed if you don't need it.
      # Refer to the commands section above for details about the implementation of this command.
      - verify_current_version:
          resource_name: "<< pipeline.deploy.component_name >>"
          namespace: "<< pipeline.deploy.namespace >>"
          current_version: "<< pipeline.deploy.current_version >>"
      # This command is used to retrieve the target revision that will be used to perform the rollback.
      # Depending on your implementation you may not need this, in which case feel free to remove it.
      # Refer to the commands section above for details about the implementation of this command.
      - retrieve_target_revision:
          resource_name: "<< pipeline.deploy.component_name >>"
          namespace: "<< pipeline.deploy.namespace >>"
          target_version: "<< pipeline.deploy.target_version >>"
      # This step will create a new deploy with PENDING status that will show up in the deploys tab in the UI
      - run:
          name: Plan release of deploy release smoke test
          command: |
            circleci run release plan  \
              --environment-name=${ENVIRONMENT_NAME} \
              --namespace=${NAMESPACE} \
              --component-name=${COMPONENT_NAME} \
              --target-version=${TARGET_VERSION} \
              --rollback
      # This command will perform the actual rollback, using the revision retrieved by retrieve_target_revision
      - perform_rollback:
          resource_name: "<< pipeline.deploy.component_name >>"
          namespace: "<< pipeline.deploy.namespace >>"
      # This step will update the PENDING deployment marker to RUNNING.
      # If you are not going to perform any validation you can just remove this.
      - run:
          name: Update planned release to RUNNING
          command: |
            circleci run release update \
              --status=RUNNING
      # This step performs validation on the deployment status after the rollback and sets the failure reason if the validation fails.
      # if you don't want to perform any validation you can just remove this.
      - validate_deployment:
          resource_name: "<< pipeline.deploy.component_name >>"
          target_version: "<< pipeline.deploy.target_version >>"
          namespace: "<< pipeline.deploy.namespace >>"
      # These last two steps update the PENDING deployment marker to SUCCESS or FAILED, based on the outcome of the job.
      - run:
          name: Update planned release to SUCCESS
          command: |
            # if the rollback failed, we don't want to update the status to SUCCESS. This is unnecessary if there is no logic around
            # validating the deployment status.
            if [ -f failure_reason.env ]; then
              exit 0
            fi
            circleci run release update \
              --status=SUCCESS
          when: on_success
      - run:
          name: Update planned release to FAILED
          command: |
            if [ -f failure_reason.env ]; then
              source failure_reason.env
            fi
            FAILURE_REASON="${FAILURE_REASON:-}"
            circleci run release update \
             --status=FAILED \
             --failure-reason="$FAILURE_REASON"
          when: on_fail
  # This job handles the cancellation of the rollback deploy marker if the rollback job is canceled
  cancel-rollback:
    docker:
      - image: cimg/aws:2023.03
    steps:
      - run:
          name: Update planned release to CANCELED
          command: |
            circleci run release update \
             --status=CANCELED

workflows:
  rollback:
    jobs:
      - rollback-component:
          context:
            # provide any required context
      - cancel-rollback:
          context:
            # provide any required context
          requires:
            - rollback-component:
              - canceled
          filters:
            branches:
              only: main

[path]

[title]

Set up a rollback pipeline

Introduction

Prerequisites

Set up a rollback pipeline

1. Start the rollback pipeline setup

2. Confirm GitHub App installation

3. Create a pipeline definition

4. Add deploy markers

5. Set up rollback config

Run a rollback pipeline

Change rollback pipeline

Configuration tips

Example rollback pipeline configuration