cmd+/

Set up release validation manually

Cloud

Release validation evaluates webhook signals from your monitoring tool during an evaluation window after each deploy. This guide shows how to configure it by editing your CircleCI configuration file, rather than through the Smart Deployments guided setup.

Introduction

The Set up Smart Deployments guided setup generates this configuration for you, but it is unavailable in two cases:

Your code is not stored in a GitHub repository.
Your organization does not have AI features enabled.

Follow this guide in either case. The result is the same release validation feature. The difference is that you write the configuration and connect your monitoring tool yourself.

What you can set up manually

Release validation and deploy markers are configuration, so they work on any version control system. Rollback and deploy pipelines are a web app feature that currently requires GitHub.

Capability Needs GitHub Notes

Capability	Needs GitHub	Notes
Deploy markers	No	Add the `circleci run release` commands to any project. See the Set up Deploy Markers Manually section.
Release validation	No	Add a `validation` block to your release job, as described on this page.
Rollback and deploy pipelines	Yes	See the Set up Rollbacks and Set up Deploys guides.
Automatic rollback	Yes	Requires a rollback pipeline, so it inherits the GitHub requirement.

Deploy markers

Add the circleci run release commands to any project. See the Set up Deploy Markers Manually section.

Release validation

Add a validation block to your release job, as described on this page.

Rollback and deploy pipelines

Yes

See the Set up Rollbacks and Set up Deploys guides.

Automatic rollback

Yes

Requires a rollback pipeline, so it inherits the GitHub requirement.

Without a rollback pipeline, release validation detects failures but cannot revert them.

A failing check still marks the release as failed, records the events that caused it, and surfaces the failure in the deploys UI. CircleCI then reports the rollback as skipped, because no rollback pipeline is configured. Treat validation as your alerting and audit trail in this case, and roll back through your own tooling.

Prerequisites

A CircleCI account connected to your code. You can sign up for free.
A CircleCI project with a workflow configured to deploy your code.
Deploy markers configured in that workflow. See the Set up Deploy Markers Manually section.
A monitoring tool that can send outbound webhooks. Datadog, Prometheus, and Alertmanager are supported directly, and you can connect any other tool as a custom provider.
Organization administrator permissions, to create a webhook secret.

1. Add a release job to your configuration

Release validation attaches to a release job, which is a job with type: release that names the release plan created by your circleci run release plan step. If your configuration already has one, skip to the next step.

The plan_name value must match the deploy name you passed to circleci run release plan. In the following example, both are my-service-release:

Deployment job and release job

version: 2.1

jobs:
  deploy-my-service:
    executor: some-executor
    steps:
      - checkout
      - run:
          name: Plan deployment
          command: |
            circleci run release plan my-service-release \
              --environment-name=production \
              --component-name=my-service \
              --target-version=<some-version-name>
      - run:
          name: Perform deployment
          command: <your-deployment-logic>
      - run:
          name: Update planned deployment to running
          command: circleci run release update my-service-release --status=running

  release-my-service:
    type: release
    plan_name: my-service-release

workflows:
  deploy-service:
    jobs:
      - deploy-my-service
      - release-my-service:
          requires:
            - deploy-my-service

For details of the circleci run release commands, see the Configure Deploy Markers guide. For the release job keys, see the Configuration Reference.

2. Turn on release validation

Add a validation block to your release job. Release validation needs only one field to turn it on:

Release job with validation enabled

jobs:
  release-my-service:
    type: release
    plan_name: my-service-release
    validation:
      enabled: true

With enabled: true and nothing else, CircleCI creates a system catch-all check. The catch-all accepts any signal from your monitoring tool, whatever its name, and fails when the criteria matches firing, triggered, or alert.

Start here. A catch-all is enough when one monitor guards one component, and it saves you from matching signal names before you have confirmed the webhook arrives at all.

3. Add named validation checks

Move to named checks when one catch-all is too blunt, for example when you want a latency monitor and an error rate monitor to fail the release independently.

Each entry under validation.webhooks defines one check. As soon as you add any entry, CircleCI stops using the catch-all, and every monitor must send a signal_name matching the name of the entry that handles it.

Release job with two named checks

jobs:
  release-my-service:
    type: release
    plan_name: my-service-release
    validation:
      enabled: true
      webhooks:
        - name: error_rate
          provider: datadog
          max_failures: 2

        - name: latency_smoke
          provider: datadog

This example relies on the Datadog defaults for data_points and fail_when. Add explicit mappings when your monitoring tool sends payloads in a shape those defaults do not understand.

For every field, the data_points keys, and the provider defaults, see the Release Validation Configuration Reference page.

4. Set the evaluation window

The engine waits 20 minutes for signals by default. Set evaluation_time to change how long it waits after a deploy completes:

Release job with an explicit evaluation window

jobs:
  release-my-service:
    type: release
    plan_name: my-service-release
    validation:
      enabled: true
      evaluation_time: 30m
      webhooks:
        - name: error_rate
          provider: datadog

The default window is 20 minutes. Match it to how quickly your monitors fire: a window shorter than your monitor evaluation interval closes before any signal can arrive, so the release passes without ever being checked.

If you have a rollback pipeline configured, add auto_rollback_on_failure: true to revert automatically when a check fails. On a version control system other than GitHub, leave it out: there is no rollback pipeline for it to trigger.

5. Connect your monitoring tool

This step is required. Without it, the engine receives no signals, so every release passes without being checked.

5.1. Copy the webhook ingest URL

All validation webhooks POST to the following URL:

https://circleci.com/api/v3/deploy/hooks/<org-id>/validate

Replace <org-id> with your organization ID. The URL is the same for every webhook secret in your organization. You can also copy the full URL, with your organization ID already filled in, from your organization deploy settings.

To find your organization ID and organization slug, select Org in the CircleCI web app sidebar to open your organization settings page.

The organization ID is available to copy from the org settings overview page.

screenshot showing where to find your organization ID

Figure 1. Organization ID available in organization settings

5.2. Create a webhook secret

CircleCI authenticates validation webhooks with a bearer token. To create one, follow these steps:

In the CircleCI web app, select your org from the org cards on your user homepage.
Select Org from the sidebar to open your organization settings page.
Select Deploys.
Find the Webhook secrets card.
Select Create webhook secret.
Enter a descriptive name, for example Production Datadog.
Optionally set an expiration date.
Copy the plaintext token before you close the modal.

The plaintext token is shown once. Store it in your monitoring tool or your secrets manager straight away. If you lose it, revoke the secret and create a new one.

5.3. Authenticate webhook requests

Configure your monitoring tool to send the following custom header on every validation webhook:

{
  "Authorization": "Bearer <secret>"
}

Replace <secret> with the token you copied. Do not include the angle brackets.

5.4. Configure your monitoring provider

Follow the instructions for your monitoring tool. Each provider needs the ingest URL, the bearer auth header, and a set of tags that let CircleCI match the alert to the right release.

Configure Datadog

In Datadog, create or edit a webhook integration for alerts.
Set the webhook URL to the CircleCI ingest URL.
Add the bearer auth custom header.

Set the payload to include the provider, the alert transition, and your tags:

{
  "provider": "datadog",
  "alert_transition": "$ALERT_TRANSITION",
  "tags": "$TAGS"
}

Ensure your alert notifications include the tags listed below on every payload. Use the format tag_name:value in the tags array.

Table 1. Datadog tags CircleCI matches on
Tag	Required	Purpose
`component_name`	Yes	The component for the release under validation.
`env`	Yes	The environment, for example `production`.
`signal_name`	Yes, when using named webhooks	Must match a `validation.webhooks[].name` value in your configuration. This routes the alert to the correct check when a release has more than one check.
`version`	No	The deployed version. Omit this tag only if you accept fan-out to all matching running releases.
`project_id`	No	The CircleCI project UUID. Omit this tag to match any project that shares the same component and environment.
`namespace`	No	The component instance namespace. Omit this tag to match any namespace for that component and environment.

Map your Datadog template variables so that component_name, version, env, and signal_name align with the values your deploy markers use.

For Datadog, the engine also accepts the monitor tags app:<name> and service:<name> as aliases for component_name, and environment:<name> as an alias for env. Prefer the canonical names.

Configure Prometheus or Alertmanager

In Prometheus or Alertmanager, create or edit a webhook receiver.
Set the URL to the CircleCI ingest URL.
Add the bearer auth custom header.
Include the keys listed below under groupLabels in the webhook payload.

Table 2. Prometheus and Alertmanager labels CircleCI matches on
Label	Purpose
`alertname`	The validation signal name. This must match a `validation.webhooks[].name` value in your configuration.
`component_name`	The component for the release under validation.
`env`	The environment.
`version`	The deployed version.

Check that your payload shape matches the following example:

{
  "provider": "prometheus",
  "status": "firing",
  "groupLabels": {
    "alertname": "error_rate",
    "component_name": "my-service",
    "env": "production",
    "version": "1.0.0"
  }
}

Configure a custom monitoring tool

Configure an alert webhook in your monitoring tool.
POST to the CircleCI ingest URL with the bearer auth custom header.
Send a JSON body that includes the fields listed below.

Table 3. Fields to include in a custom webhook payload
Field	Purpose
`provider`	Set this to `custom`.
`component_name`	The application or component name.
`env`	The environment.
`signal_name`	The validation signal. This must match a `validation.webhooks[].name` value in your configuration.
`version`	The deployed version.
`alert_transition`	The condition signal, for example `triggered`.

Check that your payload shape matches the following example:

{
  "provider": "custom",
  "component_name": "my-service",
  "version": "1.0.0",
  "env": "production",
  "signal_name": "error_rate",
  "alert_transition": "triggered"
}

If your monitoring tool sends payloads in a different shape, map the fields explicitly in your configuration. See the Data Point Keys section.

6. Verify your setup

Confirm the two halves work in order. Check that validation starts, then check that your webhook arrives.

Trigger your deploy workflow and let it complete.
Open the deployment from the deploys timeline and find the Release Validation panel. It shows an evaluation deadline and each check in a Listening state. If the panel is absent, validation is not running: check that your release job has validation.enabled: true and that plan_name matches your circleci run release plan deploy name.
Trigger a test alert in your monitoring tool while the window is open.
Refresh the deployment page. The event appears under the matching check, with a note saying whether it matched the fail criteria.

An event that arrives but attaches to no check means the payload reached CircleCI and the matching failed. See the next section.

Troubleshooting

If your webhook produces no event on the deployment, work through the following causes.

Cause How to check

Cause	How to check
The request is not authenticated	Your monitoring tool reports a delivery failure. Confirm the `Authorization` header is `Bearer <secret>` with a token that is neither revoked nor expired, in the Webhook secrets card of your organization deploy settings.
The component name does not match	The extracted `component_name` must match the component on the release exactly. Confirm your monitor sends `component_name`, or a Datadog alias such as `app`, with the same value as `--component-name` in your deploy marker.
The environment does not match	`env` is not optional for matching. Confirm your monitor sends the same value as `--environment-name` in your deploy marker.
The signal name does not match	Once you add entries under `validation.webhooks`, each monitor must send a `signal_name` matching an entry name. A missing or unmatched name is ignored. Remove your `webhooks` entries to fall back to the catch-all and confirm delivery, then add them back.
The criteria expression does not match	CircleCI normalizes criteria values to lowercase before evaluation. Write lowercase literals in `fail_when`, so `"triggered"` rather than `"Triggered"`.
The window closed before the alert	Compare the alert time with the evaluation deadline on the deployment. If the monitor fires more slowly than the window, raise `evaluation_time`.

The request is not authenticated

Your monitoring tool reports a delivery failure. Confirm the Authorization header is Bearer <secret> with a token that is neither revoked nor expired, in the Webhook secrets card of your organization deploy settings.

The component name does not match

The extracted component_name must match the component on the release exactly. Confirm your monitor sends component_name, or a Datadog alias such as app, with the same value as --component-name in your deploy marker.

The environment does not match

env is not optional for matching. Confirm your monitor sends the same value as --environment-name in your deploy marker.

The signal name does not match

Once you add entries under validation.webhooks, each monitor must send a signal_name matching an entry name. A missing or unmatched name is ignored. Remove your webhooks entries to fall back to the catch-all and confirm delivery, then add them back.

The criteria expression does not match

CircleCI normalizes criteria values to lowercase before evaluation. Write lowercase literals in fail_when, so "triggered" rather than "Triggered".

The window closed before the alert

Compare the alert time with the evaluation deadline on the deployment. If the monitor fires more slowly than the window, raise evaluation_time.

For what each field means during matching, see the Data Point Keys section.

Next steps

Review every field you can set on the validation block. See the Release Validation Configuration Reference page.
Understand what happens when validation passes or fails. See the How Release Validation Works section.
If your code is in a GitHub repository, set up a rollback pipeline to enable automatic rollback. See the Set up Rollbacks guide.

[path]

[title]