Creating integrations, dashboards, notifications, and more using CircleCI webhooks
Developer Advocate
Webhooks allow for communication between services and APIs, which makes them the glue of our interconnected, cloud-based application environment. If you are familiar with APIs, you can learn to use webhooks. CircleCI offers a webhooks feature for our CI/CD platform that lets you subscribe and react to CircleCI events such as workflow and job completed.
This tutorial showcases the webhooks feature and gives you steps for getting started. It is intended for folks who are already using CircleCI for building projects. There is also a free sample NodeJS webhook consumer project.
How are webhooks different from APIs?
The main difference between webhooks and APIs is that, for a developer, APIs are a pull experience and webhooks are a push experience. A client requests data from the API. If a client wants more data, or needs to check whether that data has updated, it makes another request. With a webhook, a service (client) is set up as a webhook endpoint, and you give that endpoint to another service. That other service (in our case CircleCI), calls your webhook endpoint when an event happens that you are interested in. There are many uses for webhooks, from GitHub integrations to Discord chat bots, Slack notifications, and so on. There are hundreds and hundreds of possibilities.
CircleCI allows you to set up webhooks for workflow and job completion events in a project. These events trigger when each has finished, either successfully or unsuccessfully.
Without webhooks, this is only possible by polling the CircleCI APIs repeatedly. To use the APIs this way, the service had to make repeated requests to get the freshest data. This method is not ideal for development teams, who had to maintain a service that made hundreds (if not thousands) of requests per day. Repeated API requests are not great for CircleCI either, because we had to serve all those thousands of requests.
What use cases do webhooks unlock?
In a CI/CD context, webhooks let you build all kinds of integrations like dashboards, data logging, and project management tools. Webhooks enable your team to automate notifications to downstream clients, to inform them of a new version they need to update to, for example. You can set up webhooks either on a single project or for multiple projects and then aggregate their events. You can then process them in your backend in the way that works for your use case.
Setting up a CircleCI webhook
To set up a new webhook on an individual project using the CircleCI web interface, go to the project settings for your CircleCI project. Select the Webhooks section, then click the Add Webhook button.
The options you can pass at webhook creation are:
- Name
- URL of the endpoint the webhook will be sent to
- Which events to send
- Certificate validation
- Secret value
In the example image we are using the temporary URL generated by the Ngrok tunnel from a local server, and a secret token of super-secret-1234. Of course, this is just an example to illustrate a point; your secret will be different.
Clicking Save webhook turns it on. CircleCI will send all workflow completed events to the endpoint you entered in the URL field.
Handling the payload and responding to the webhook
The webhook sends the payload as the POST request body, to the endpoint you specified when setting it up.
The server handling the webhook must send a 200 status code in response, or the webhook may be resent multiple times, which can lead to duplicate events on your end. It is best practice to pass all incoming webhook payloads to a message queue like RabbitMQ or Amazon SQS, and process them asynchronously.
What is in the webhook payload?
The payload contains information on the workflow that has completed, including its name, status, timings, its project and pipeline, the commit it is associated with, and the trigger that began the pipeline execution. The value of pipeline trigger will be webhook for VCS commit requests. The other two options are: api and schedule.
Here is an example of a full workflow payload:
{
type: 'workflow-completed',
id: 'f85a2fd5-108d-38e2-a990-fb49e7001515',
happened_at: '2021-06-01T11:23:14.146Z',
webhook: {
id: '1154a973-e434-407c-8051-9259d8a53e99',
name: 'Workflow Completed'
},
workflow: {
id: '89b02ea5-7fca-4017-a902-747e68235ffd',
name: 'do-all-the-things',
status: 'success',
created_at: '2021-06-01T11:23:07.441Z',
stopped_at: '2021-06-01T11:23:13.942Z',
url: 'https://app.circleci.com/pipelines/github/zmarkan/very-simple-non-app/26/workflows/89b02ea5-7fca-4017-a902-747e68235ffd'
},
pipeline: {
id: 'ac0cc08f-67a2-4adb-9fa6-db633e390670',
number: 26,
created_at: '2021-06-01T11:23:07.371Z',
trigger: { type: 'webhook' },
vcs: {
provider_name: 'github',
origin_repository_url: 'https://github.com/zmarkan/very-simple-non-app',
target_repository_url: 'https://github.com/zmarkan/very-simple-non-app',
revision: 'f1e37011b5ff2d1703d4de9143c52ba650acae53',
commit: [Object],
branch: 'streamline'
}
},
project: {
id: 'e5c5dce3-7ef8-4227-9888-6b2b97b0a5ab',
name: 'very-simple-non-app',
slug: 'github/zmarkan/very-simple-non-app'
},
organization: { id: '8995ddc0-b07a-4bc2-8eb6-816c67a512fa', name: 'zmarkan' }
}
All the objects in the webhook contain unique id fields. You can use an id to query the CircleCI APIs if you need more context from CircleCI.
Validating the webhook signature
Your webhook endpoints can receive events from anywhere on the web, not just CircleCI, by whomever knows the endpoint address. You cannot control where requests are coming from, so it is important to validate the integrity of the event senders. You need to make sure you are not letting in malicious requests.
To make sure requests are safe, webhooks allow secret validation. When you set up a new webhook, you can pass in a secret value. In the sample, we used super-secret-1234. CircleCI can use that to create a HMAC SHA256 digest of the webhook request body. The secret digest is in a circleci-signature header when the webhook payload is received. The value will be v1 parameter of the form: circleci-signature: v1=YOUR_HMAC_SHA256_DIGEST.
You can validate the request by creating a digest of the request body of your own with the same secret key and same algorithm, and then comparing them to the one in circleci-signature header. If both values are the same, you can assume it is really from CircleCI. A non-matching signature could mean it was sent by a malicious actor.
Here is an example snippet from the sample NodeJS Express app:
// req, res are Express objects
let signature = req.headers["circleci-signature"].substring(3)
const key = "super-secret-1234" // Same string as used in webhook setup
let testDigest = crypto.createHmac('sha256', key).update(JSON.stringify(req.body)digest('hex')
if (testDigest !== crypto.sign) {
console.log("Webhook signature not matching")
console.log(`Signature: ${signature}`)
console.log(`Test digest: ${testDigest}`)
res.status(403).send("Invalid signature")
}
Creating a dashboard integration using webhooks
Once you are able to receive and process webhooks, you can build many types of integrations with them. I have created a sample dashboard that shows the status of workflows as they are coming in using Vue and Express.
You can find the source code on GitHub. The project I used for testing webhooks is also available on GitHub.
Conclusion
CircleCI webhooks are a powerful way to integrate CircleCI pipelines with the rest of your business. From dashboards to project tracking to notifications, webhooks provide many possibilities for creating new ways to get and share information. In this tutorial, you have learned how to get started setting up a webhook for a project that notifies you when a workflow has finished. Now that you have learned about some best practices for handling webhooks and ensuring system security, I hope you and your team find many ways to use them in your own projects.
If you have any feedback or suggestions about topics I should cover next, reach out to me on Twitter - @zmarkan.
