Triggering pipelines from pipelines: Orchestrating complex CI/CD workflows with CircleCI

It’s no secret that software development is becoming an increasingly complex process. The individual elements of software like apps, libraries, and services are interconnected and dependent on many other elements. Development teams deal with a whole ecosystem of services that they develop, maintain, or depend on, which in turn are dependent on other software ecosystems, maintained by separate teams. Maintaining this ecosystem is as complex as you might imagine. Making sure the whole system works, and keeps working as intended, is a hugely challenging effort.

Triggering CI/CD pipelines from other pipelines is one of many techniques that can help you manage complex connected and interdependent development projects. In this tutorial, I will cover some use cases for this technique and why it can be helpful. You can follow along as I present an example of how to implement a pipeline-to-pipeline trigger using CURL.

Prerequisites

This intermediate to advanced tutorial requires some understanding of CircleCI and its pipelines. It will be especially useful if you use CircleCI and have some projects that have already been configured. Everything covered in this article will work on all CircleCI plans.

Use cases for pipeline-to-pipeline triggers

There are several reasons to consider triggering new pipelines from existing pipelines. Microservices orchestration is a prime example. It is a common pattern to have microservices in each respective version control repository, each of which requires a separate CircleCI pipeline configuration. As one service gets deployed, you could redeploy others that depend on this new version. Or you could trigger an expansive suite of integration tests that run independently of the wider test and deployment flow.

If you are not using microservices, you can use pipeline-to-pipeline triggers to publish and test libraries that are integrated with downstream consumers.

Implementing pipeline triggers

One workflow for triggering pipelines follows these steps:

1. The first pipeline is executed, makes a build, runs some tests, and triggers a deployment.
2. After that process completes successfully, the first pipeline makes an API call to CircleCI to trigger another pipeline in a different project.
3. The second pipeline executes a build, runs tests, and triggers a deployment.

In this scenario, the first pipeline can also pass some pipeline parameters like the name and version of what was deployed, so that the newly triggered pipeline has some context.

Coordinating separate services using pipeline triggers

To orchestrate two separate services, you can invoke the CircleCI API from a single job. You will need to know the name of the project with the pipeline you want to trigger from that job. To address a pipeline with an API, you will need:

• The organization name (zmarkan in my case)
• The project name (pinging-me-softly because I have an odd sense of humor)
• The pipeline definition ID for the pipeline you want to trigger. You can find this ID in Project Settings > Pipelines.

Next, get a personal API token. You must have push access to both projects for this to work. You can store that API token in this project’s environment variable. For increased security, store it in a context. In my example, I used a context called circleci-api, with a CIRCLE_API_TOKEN name for that environment variable.

To trigger a pipeline, you will need to send a POST request to the pipeline’s endpoint of the project:

  trigger-new-pipeline:
      docker: 
        - image: cimg/base:2021.11
      resource_class: small
      steps:
        - run:
            name: Ping another pipeline
            command: |
              curl --request POST \
                --url https://circleci.com/api/v2/project/gh/zmarkan/pinging-me-softly/pipeline/run \
                --header "Circle-Token: $CIRCLECI_API_KEY" \
                --header "content-type: application/json" \
                --data '{"definition_id":"2338d0ae-5541-4bbf-88a2-55e9f7281f80","config":{"branch":"main"},"checkout":{"branch":"main"},"parameters":{"image-name":"'$DOCKER_LOGIN"/"$CIRCLE_PROJECT_REPONAME":"0.1.<< pipeline.number >>'"}}'

To trigger the pipeline, there is a single command to run: curl. The API key is passed in the Circle-Token header, passing in the environment variable that contains the token. The payload body is a JSON object, which contains a branch and parameters:

{
  "definition_id": "2338d0ae-5541-4bbf-88a2-55e9f7281f80",
  "config": {
    "branch": "main"
  },
  "checkout": {
    "branch": "main"
  },
  "parameters": {
        "image-name":"'$DOCKER_LOGIN"/"$CIRCLE_PROJECT_REPONAME":"0.1.<< pipeline.number>>'"
    }
}

The value of the image-name parameter is a string constructed of three variables:

• $DOCKER_LOGIN is an environment variable set in this project or in a context.
• $CIRCLE_PROJECT_REPONAME is a built-in environment variable that specifies this project in CircleCI.
• pipeline.number is another built-in environment variable.

As a whole, this value provides the name of the Docker image we built in the previous step.

Handling the pipeline trigger

The pipeline trigger is handled for you automatically by CircleCI, the same way it handles a new git commit in your repository. By default, all your workflows will run, and it will take the last commit of a branch or tag that you specified in the API call. For this tutorial, I specified main.

You can handle any parameters you pass (image-name in my case) in your jobs and workflows. Just make sure to declare them in the parameters section of the config:

  version: 2.1

  parameters:
    image-name:
      type: string
      default: "Not a real image name"

  jobs:
    print-image-name:
      docker: 
        - image: cimg/base:2021.11
      steps:
        - checkout
        - run:
            name: Echo Image name
            command: echo << pipeline.parameters.image-name >>

  workflows:
    run-workflow:
      jobs:
        - print-image-name

This pipeline example has a single workflow. It prints out the name of the image we built in the previous pipeline and passed via the API to this pipeline.

Conclusion

In this article I have demonstrated how to use the CircleCI API to trigger another pipeline when the first project’s pipeline is complete. It is a straightforward process that requires a single API call from a running job, and is available to all CircleCI users and organizations.

This technique opens up new possibilities for orchestrating complex workflows, including microservices architecture, and decoupling expensive integration testing work from build and deploy scenarios. There are many, many more possibilities for you and your team to experiment with.

Ready to orchestrate smarter pipelines? Sign up for CircleCI and start building powerful, automated workflows across your projects, with just a few lines of config.