Developing a pipeline-builds logging system with CircleCI webhooks and Airtable Automations

Ever since CircleCI introduced webhooks, I have been excited about the possibilities this new way of integration opens up to developers. I decided to try out one of the use cases described in the webhooks documentation. This use case involves transmitting information about build-pipeline workflows into an Airtable database. The data piped into Airtable forms a log for you to monitor your workflows and you can go as far as designing graphs and other visualizations to analyze the build data.

In this tutorial, I will guide you through the process of building this set up with CircleCI webhooks and Airtable Automations.

Prerequisites

To follow along with this tutorial, you will need:

1. A CircleCI project
2. Airtable account

With these in place, you can begin.

Creating an Airtable database

The first step is to create an Airtable database. Head over to your Airtable account and create a new Airtable Base inside a workspace.

Click the base to open the database table. On the table, create the following fields using the Single line text field type:

• ID: The id of the webhook
• Project: The name of the CircleCI project triggering the webhook
• Status: The status of the project build
• Time: The time the webhook was fired

Note: You can edit or delete the fields created by default and create the new ones I just listed.

This is the subset of the webhook payload you will be capturing in Airtable.

Setting up webhook automation in Airtable

To capture webhook payloads in Airtable, you need to set up a custom Airtable Automation on the base you just created. Click on Automations at the top right-hand corner of the screen.

On the automations page, click Create Custom Automation.

This opens the custom automations page. On this page, rename the automation to Builds Automation (or any name you like). Click Add Trigger and select the When webhook is received option.

This displays a panel on the right side that shows you the endpoint where Airtable will be receiving your webhooks. Click the Copy button to copy this URL for use in the next step.

Now, you need to test this webhook endpoint to ensure that Airtable can successfully receive your CircleCI webhooks on it. To use the webhook payload in your automation setup, you need to test this endpoint with the exact payload structure you are expecting.

To send a test webhook request, you will be making use of ReqBin. Paste your Airtable webhook endpoint in the address field and select POST from the request method dropdown. Click the Content tab and paste in this sample request:

{
  "id": "3888f21b-eaa7-38e3-8f3d-75a63bba8895",
  "type": "workflow-completed",
  "happened_at": "2021-09-01T22:49:34.317Z",
  "webhook": {
    "id": "cf8c4fdd-0587-4da1-b4ca-4846e9640af9",
    "name": "Sample Webhook"
  },
  "project": {
    "id": "84996744-a854-4f5e-aea3-04e2851dc1d2",
    "name": "webhook-service",
    "slug": "github/circleci/webhook-service"
  },
  "organization": {
    "id": "f22b6566-597d-46d5-ba74-99ef5bb3d85c",
    "name": "circleci"
  },
  "workflow": {
    "id": "fda08377-fe7e-46b1-8992-3a7aaecac9c3",
    "name": "build-test-deploy",
    "created_at": "2021-09-01T22:49:03.616Z",
    "stopped_at": "2021-09-01T22:49:34.170Z",
    "url": "https://app.circleci.com/pipelines/github/circleci/webhook-service/130/workflows/fda08377-fe7e-46b1-8992-3a7aaecac9c3",
    "status": "success"
  },
  "pipeline": {
    "id": "1285fe1d-d3a6-44fc-8886-8979558254c4",
    "number": 130,
    "created_at": "2021-09-01T22:49:03.544Z",
    "trigger": {
      "type": "webhook"
    },
    "vcs": {
      "provider_name": "github",
      "origin_repository_url": "https://github.com/circleci/webhook-service",
      "target_repository_url": "https://github.com/circleci/webhook-service",
      "revision": "1dc6aa69429bff4806ad6afe58d3d8f57e25973e",
      "commit": {
        "subject": "Description of change",
        "body": "More details about the change",
        "author": {
          "name": "Author Name",
          "email": "author.email@example.com"
        },
        "authored_at": "2021-09-01T22:48:53Z",
        "committer": {
          "name": "Committer Name",
          "email": "committer.email@example.com"
        },
        "committed_at": "2021-09-01T22:48:53Z"
      },
      "branch": "main"
    }
  }
}

This sample payload has the same structure as the one CircleCI will be sending to Airtable for a workflow-completed event.

Your request should be similar to the previous screenshot.

Now, click Send to test your webhook endpoint. A successful test should return a success message.

Return to Airtable, and go to the screen where you copied the webhook endpoint. Airtable may have automatically detected your webhook request and displayed a success message. If there is no success message, press the Test trigger button.

Airtable now understands the structure of your webhook payload and you can use it within your automation.

Adding an automation action

The next step is to add an action to be performed when a webhook is received from CircleCI. Go to the Automations page. Click the + Add Action button on the Run actions tab. Select Create Record from the dropdown to create a new entry in Airtable for each webhook received. On the Properties panel to the right of the screen, select Table 1 for the Table field (this is the default name for the table created with your Airtable base).

In the Fields section of the form, select the ID field. This displays a text box underneath the field. Click the plus + icon. Then click Continue on the Webhook (from Step 1: When webhook received) option. Click body then select the id property by clicking Insert.

Use this same method to map the following table fields to their respective payload properties:

• Project: body > project > name
• Status: body > workflow > status
• Time: body > happened_at

Your fields should be similar to the ones in the next screenshot.

Turn on automation using the toggle button beside the name.

Go back to the table view. Your automation will be listed on the right side of the screen. The description should say: “When a webhook is received create a record”

Creating a webhook on CircleCI

The next step is to create your webhook on CircleCI. When your project build or workflow is complete, a webhook request is sent to Airtable to log information about the build process.

With your webhook URL ready, you can now set up a webhook on CircleCI. Go to any of your CircleCI projects and navigate to Project Settings > Webhooks. On the Webhooks page, click the Add Webhook button.

This displays a form you can use to create your webhook. Fill the form as shown in the next screenshot. Give it a name and enter your webhook endpoint URL from Airtable.

Click the Add Webhook button to create your webhook. Once created, it will be displayed in the list of webhooks.

Testing the builds-logging system

You now have everything set up to log your build information into Airtable. The time has come to test it. Run your build to completion a couple of times. Once you have a few builds that have been completed on your project, check the table on your Airtable base to review the entries.

Conclusion

And there you have it, a logging system for your CircleCI builds using CircleCI webhooks and Airtable automations. Of course, you can expand the amount of information captured by adding more fields to the table and mapping them to their respective payload properties. You can also create different views in Airtable using the data captured. With a little effort, you use this system to perform analytical operations to derive actionable information from your data.

Happy coding!