Troubleshooting CircleCI webhooks

CircleCI webhooks open up a variety of exciting use cases, from data logging and integrations with third-party monitoring and observability solutions to setting up your own custom dashboards to monitor pipeline health. To ensure that you can properly monitor events, resolve authentication errors, and also access the information contained within them, you need a reliable process to debug any errors you might encounter.

In this tutorial, I will show you how to set up a webhook on CircleCI as well as techniques for troubleshooting common errors with webhook connections so that you can be confident about the work you and your team are doing.

Prerequisites

To begin, there are a few things you need to have or set up, including:

• A CircleCI account and associated project
• Node.js installed on your system to run the sample project
• A publicly accessible URL to the API endpoint
• A text editor for editing code

With this setup, you have an environment in which you can conveniently troubleshoot your CircleCI webhooks.

Cloning and running a demo API

To begin, you will be cloning a sample Node.js API. This API receives and logs a subset of your CircleCI webhook information in an in-memory database. I have intentionally introduced an authentication error and a not found error into this application. These are the types of errors you will be debugging and fixing as you follow along with the tutorial.

Clone the project repository by running this command:

git clone --single-branch --branch base-project https://github.com/coderonfleek/circleci-webhooks-api

Navigate to the root of the project and install the required dependencies by running these commands:

cd circleci-webhooks-api
npm install

When the installation is complete, run the Node.js server with this command:

npm start

This will boot up the API application and print a message to the screen indicating that the API is now running and listening for connections on port 1337.

We are using two endpoints in this project:

• /log-circleci-webhook is the endpoint that will be receiving the CircleCI webhook and logging it into an in-memory database. It logs a simple object containing a subset of the information from the webhook payload.
• The /fetch-webhooks-logs endpoint can be called to retrieve a collection of the logged webhook data.

Troubleshooting CircleCI webhooks with Hookdeck CLI

Troubleshooting webhooks locally requires that you have a publicly accessible URL. However, locally running APIs do not have publicly accessible endpoints. You need a way to make your local endpoints publicly accessible and that can be done by using an HTTP request tunneling system.

An HTTP request tunneling system helps tunnel your webhooks into your local development environment to target an endpoint on your local API.

There are many open-source tools that can help you achieve this. The one that I have found most convenient is the Hookdeck CLI. Hookdeck CLI not only provides you with a publicly accessible URL that points to your local API endpoint, but it also comes with reporting, an event page to view your headers and payload, and other tools that make debugging webhooks less frustrating.

You can run this command to install the CLI tool for macOS:

brew install hookdeck/hookdeck/hookdeck

If you are using the Windows operating system, use this command to install the CLI tool:

scoop bucket add hookdeck https://github.com/hookdeck/scoop-hookdeck-cli.git
scoop install hookdeck

For Linux users, you can follow the instructions for installing the tool on Linux here.

Getting a CircleCI webhook URL

The next step is to use the CLI to generate a webhook URL that points to the running API application. Run this command:

hookdeck listen 1337

This command starts an interactive session where the CLI collects information about the endpoint you are about to create. Answer the questions as documented in the following list. Make sure to press Enter after each answer.

• Q: What should be your new source label?
  • A: CircleCI
• Q: What path should the webhooks be forwarded to (i.e.: /webhooks)?
  • A: /log-circleci-webhook
• Q: What’s the connection label (i.e.: My API)?
  • A: My CircleCI Webhooks Server

The CLI uses the information you enter to generate the URL. When the process is complete, the URL is printed to the screen. The CLI then indicates that it is ready to receive requests.

fikayo $ hookdeck listen 1337
🚩 Not connected with any account. Creating a guest account...
? What should be your new source label? CircleCI
? What path should the webhooks be forwarded to (ie: /webhooks)? /log-circleci-w? What's your connection label (ie: My API)? My CircleCI Webhooks Server

Dashboard
👤 Login URL: https://api.hookdeck.com/signin/guest?token=13cpjezttzt1ll5w0jb1403lj8sd8t5j2dbfv2di6k1kpshgop
Sign up in the dashboard to make your webhook URL permanent.

👉 Inspect and replay webhooks: https://dashboard.hookdeck.com/cli/events

circleci Source
🔌 Webhook URL: https://events.hookdeck.com/e/src_A345Rx6enuPRpUFuc526wiua

Connections
my-circleci-webhooks-server forwarding to /log-circleci-webhook

> Ready! (^C to quit)

Note: You will need to use the guest link in the console to access the dashboard.

Copy and paste Login URL into your browser to begin a guest login session.

Make sure that the connection status on the side menu shows Connected.

Setting up the CircleCI webhook

With your webhook URL, you can now set up a webhook on CircleCI. Go to any of your CircleCI projects and navigate to Project Settings > Webhooks. On the Webhooks page, click Add Webhook.

Fill out the fields on the form like this:

• Webhook name: Enter a descriptive name for your webhook. Use something simple like Log API Webhook.
• Receiver URL: Paste the webhook URL from the Hookdeck CLI output here.
• Secret token: This is a security feature that allows you to verify your webhook source (more on this later). Enter the value ABCD123 here.
• Certificate verification: If this is checked, the SSL certificate on your API is verified to be authentic and up to date. Click this box.
• Events: Check the Workflow Completed event.

Click Add Webhook to create your webhook. It will be displayed in the list of available webhooks.

Next, confirm that you can now receive webhooks from your CircleCI project. To run the project build either:

• Make a new commit
• Click Rerun workflow from start

Once the build is complete, go to the terminal where your Hookdeck CLI session is running. You should have a new entry.

The entry on the CLI confirms that you are successfully receiving the webhook. But there is a 500 server error.

The webhook entry on the CLI consists of 4 elements in this order:

1. HTTP status code (500)
2. Request method (POST )
3. Endpoint that you configured the CLI to route your webhooks to (/log-circleci-webhook)
4. Event page URL where you can view details about your webhooks

Go on to the next section to fix the 500 error.

Troubleshooting the authentication error

Before making any assumptions, inspect the server response using the event’s page. Copy the event page link from the Hookdeck CLI session and load it in your browser.

On this page, you can inspect all the details about the webhook you just received. Take a minute to review the Headers section.

Review the webhook payload in the Body section.

Scroll down to the Attempts section. Click the red status code badge to review the server response details.

The response message reveals that we are failing the security check on the webhook payload. Because we defined an API secret, CircleCI sends the circleci-signature header. This header contains an encrypted version of the real payload and must be validated against the unencrypted payload received by using the secret key.

This is a security check to prevent attackers from replacing the actual payload with a malicious one that could corrupt your API.

We know that we are receiving the actual payload from CircleCI because we set everything up and triggered the webhook. We know it is not an attacker’s payload causing the authentication failure. Something else must be wrong with our set up.

Our next step is to review the validation logic on our API. This can be found in the server.js file.

//Validate payload
function validatePayload(req, res, next) {
  if (req.method == "POST") {
    if (!req.rawBody) {
      return next("Request body empty");
    }

    const sig = Buffer.from(req.get(sigHeaderName) || "", "utf8");
    const hmac = crypto.createHmac(sigHashAlg, secret);
    const digest = Buffer.from(
      "v1=" + hmac.update(req.rawBody).digest("hex"),
      "utf8"
    );

    if (sig.length !== digest.length || !crypto.timingSafeEqual(digest, sig)) {
      return next(
        `Request body digest (${digest}) did not match ${sigHeaderName} (${sig})`
      );
    }
  }

  return next();
}
app.use(validatePayload);

Three variables being used here in the code are worth checking out to confirm that they are referencing the correct values. These are:

• sigHeaderName, which represents the signature header name sent by CircleCI.
• sigHashAlg, which is the algorithm used for the encryption.
• secret, which is the API secret set on our CircleCI webhook.

These values are set on lines 10-12 in the server.js file. Here is the code for these definitions:

const sigHeaderName = "circleci-signature";
const sigHashAlg = "sha256";
const secret = "XXX-XXX";

Did you catch the error? If the secret is not the same as the one we set in our webhook form, the validation will always fail.

CircleCI does not allow you to view the API secret you set in your webhooks a second time; you can only reset it. This is why you need to make sure that you remember the value or store it somewhere secure.

For this tutorial, we know we set the secret as a simple ABCD123 string. In real-world applications, you will want to set a more complicated secret and reference it from an environment variable in your code.

Change the value of the secret to the correct value, save the file, and restart the server.

To test the fix, you need to trigger a new webhook. You can trigger another build on your CircleCI project. If you do not want to wait for a new build, you can use the Retry button to rerun the webhook. The Retry button is on the top of the event page beside the Bookmark button.

After you click Retry, a new request will be listed in the Attempts section of the event page.

There will also be a new webhook entry on the CLI.

Good work! We have cleared the 500 error, but now we have a new error. This 404 not found error indicates that our specified endpoint (/log-circleci-webhook) cannot be found.

Troubleshooting the “not found” error

Our second webhook attempt resulted in a 404 error on the destination endpoint. This tells us that even though we are successfully receiving our webhooks, the endpoint is not at the specified location, or it does not exist. 404 errors can be fixed, most of the time, by checking for typos or misspellings in the specified route name. Worst-case scenario, the route truly does not exist and needs to be created.

Click the red 404 badge in the Attempts section to review the server response for this error.

This shows the actual response from the running server, and it confirms that the specified endpoint cannot be found.

The endpoint specified for the webhook to hit is /log-circleci-webhook, which can be found in the routes.js file at the root of the project directory.

router.post("/log-circleci-hook", async function (req, res) {
  //console.log(req.body);

  const payload = req.body;

  let webhook_info = {
    name: payload.webhook.name,
    project: payload.project.name,
    workflow: payload.workflow.name,
    status: payload.workflow.status
  };

  const save_webhook = await req.db
    .collection("webhooks")
    .insertOne(webhook_info);

  res.status(201).send({
    message: "Webhook Event successfully logged"
  });
});

Closer review of the route handler reveals a typo. On line 10 webhook is incorrectly spelled as hook. Correct this misspelling and save the file.

You will need to restart the Node.js server to allow the changes to take effect. Shut down the server using Ctrl + C and restart it with the npm start command.

Click the Retry button again and watch the Attempts section for a new webhook request. The successful webhook will be indicated by the 201 status code.

There will also be a webhook entry on your Hookdeck CLI session.

We have a successful operation on our API.

In the Attempts section, click the green status badge to show the server response. The server returns a success message.

With a successful operation on our API, our webhook should be logged. Visit the endpoint /fetch-webhooks-logs to review the collection.

Conclusion

Just like any other architectural component of an application, webhooks can run into errors. Debugging is a part of the workflow when developing applications or setting up application infrastructure. Errors are inevitable, so knowing what to look out for when a certain error occurs and having the right tools to debug them makes troubleshooting faster and much easier.

In this article, we have demonstrated how to debug CircleCI webhooks in a way that gives clear insight into the errors taking place.

Happy coding!