Advanced pipeline orchestration with the circleback pattern
Developer Advocate
An alternative approach would be to keep the first pipeline running, checking periodically whether the second pipeline has finished. The main drawback of this approach is the increased cost for the running job’s continuous polling.
Depending on the status of the second pipeline, the first continues executing until it either terminates successfully or fails.
Why have pipelines wait for other projects to complete?
As mentioned in the introduction, this is an advanced technique aimed at taming the complexity that stems from working with multiple projects. It can be used by teams working on multiple interdependent projects that they do not want to put in a single repository like a monorepo. Another use would be to have a centralized repository for tests, for example in a hardware company. This technique could also be useful for integration testing of a microservices application, or for orchestrating complex deployment scenarios. There are many possibilities.
Implementing pipeline triggers
We have 2 pipelines we want to orchestrate
- Pipeline A, which does the triggering
- Pipeline B is triggered, and circles back to pipeline A
Pipeline B is dependent on A, and can be used to validate A.
Both pipelines need to have API keys set up and available. You can use the API key set as an environment variable (CIRCLECI_API_KEY) in the job in pipeline A, and also in pipeline B when it calls back. You can either set it in both projects, or at the organization level as a context. For this tutorial, I set it at the organization level as the circleci_api context, so that both projects can use the same API key.
Trigger the pipeline
The triggering process is explained in depth in the first part of this tutorial, Triggering pipelines from other pipelines. In this follow-up tutorial, I will cover just the important differences.
- To circle back from the pipeline, pass the original pipeline’s ID to it. Then it can be retrieved and reached with the API.
- You also need to store the triggered pipeline’s ID. You will need to get its result later on.
In the sample code, the parameter is called triggering-pipeline-id:
curl --request POST \
--url https://circleci.com/api/v2/project/gh/zmarkan-demos/circleback-cicd-project-b/pipeline \
--header "Circle-Token: $CIRCLECI_API_KEY" \
--header "content-type: application/json" \
--data '{"branch":"main","parameters":{"triggering-pipeline-id":"<< pipeline.id >>"}}'
To store the pipeline ID, wrap your curl call in $() and assign it to the variable CREATED_PIPELINE. To extract the ID from the response body, use the jq tool, and write it to the file pipeline.txt:
CREATED_PIPELINE=$(curl --request POST \
--url https://circleci.com/api/v2/project/gh/zmarkan-demos/semaphore_demo_project_b/pipeline \
--header "Circle-Token: $CIRCLECI_API_KEY" \
--header "content-type: application/json" \
--data '{"branch":"main","parameters":{"triggering-pipeline-id":"<< pipeline.id >>"}}' \
| jq -r '.id'
)
echo "my created pipeline"
echo $CREATED_PIPELINE
mkdir ~/workspace
echo $CREATED_PIPELINE > pipeline.txt
Now that you have the file pipeline.txt created, use persist_to_workspace to store it and use it in a subsequent job:
- persist_to_workspace:
root: .
paths:
- pipeline.txt
The whole job configuration is here:
...
jobs:
trigger-project-b-pipeline:
docker:
- image: cimg/base:2021.11
resource_class: small
steps:
- run:
name: Ping another pipeline
command: |
CREATED_PIPELINE=$(curl --request POST \
--url https://circleci.com/api/v2/project/gh/zmarkan-demos/semaphore_demo_project_b/pipeline \
--header "Circle-Token: $CIRCLECI_API_KEY" \
--header "content-type: application/json" \
--data '{"branch":"main","parameters":{"triggering-pipeline-id":"<< pipeline.id >>"}}' \
| jq -r '.id'
)
echo "my created pipeline"
echo $CREATED_PIPELINE
mkdir ~/workspace
echo $CREATED_PIPELINE > pipeline.txt
- persist_to_workspace:
root: .
paths:
- pipeline.txt
...
Orchestrating the waits
The previous job will trigger a pipeline B, which needs to complete before it can circle back to pipeline A. You can use the approval job in CircleCI like this:
...
workflows:
node-test-and-deploy:
jobs:
...
- trigger-project-b-pipeline:
context:
- circleci-api
requires:
- build-and-test
filters:
branches:
only: main
- wait-for-triggered-pipeline:
type: approval
requires:
- trigger-project-b-pipeline
- check-status-of-triggered-pipeline:
requires:
- wait-for-triggered-pipeline
context:
- circleci-api
...
After the job trigger-project-b-pipeline, enter the wait-for-triggered-pipeline. Because that job type is approval it will wait until someone (in this case, the API) manually approves it. (More details in the next section.) After it is approved, add a requires stanza so it continues to a subsequent job.
Both jobs that use the CircleCI API have the context specified, so the API token is available to both as an environment variable.
Circle back to pipeline A
For now we are done with pipeline A, and it is pipeline B’s time to shine. CircleCI’s approval job is a special kind of job that waits until accepted. It is commonly used to hold a pipeline in a pending state until it is approved by a human delivery lead or infosec engineer.
At this point, pipeline B knows the ID of pipeline A, so you can use the approval API to get it. You only have the ID of the pipeline being run, but not the actual job that needs approval, so you will need more than one API call:
- Get all jobs in the pipeline
- Find the approval job by name
- Send a request to approve the job
Approving the job allows pipeline A to continue.
If the tests fail in pipeline B, then that job automatically fails. A workflow with required jobs will not continue in this case. You can get around by using post-steps in the pipeline, which always executes. The whole workflow is shown in the next sample code block.
Parameters:
parameters:
triggering-pipeline-id:
type: string
default: ""
...
workflows:
node-test-and-deploy:
jobs:
- build-and-test:
post-steps:
- approve-job-in-triggering-pipeline
context:
- circleci-api
Script to perform the approval API call can be implemented like this. For this tutorial, I used a command.
...
commands:
approve-job-in-triggering-pipeline:
steps:
- run:
name: Ping CircleCI API and approve the pending job
command: |
echo << pipeline.parameters.triggering-pipeline-id >>
if ! [ -z "<< pipeline.parameters.triggering-pipeline-id >>" ]
then
workflow_id=$(curl --request GET \
--url https://circleci.com/api/v2/pipeline/<< pipeline.parameters.triggering-pipeline-id >>/workflow \
--header "Circle-Token: $CIRCLECI_API_KEY" \
--header "content-type: application/json" \
| jq -r '.items[0].id')
echo $workflow_id
waiting_job_id=$(curl --request GET \
--url https://circleci.com/api/v2/workflow/$workflow_id/job \
--header "Circle-Token: $CIRCLECI_API_KEY" \
--header "content-type: application/json" \
| jq -r '.items[] | select(.name == "wait-for-triggered-pipeline").id')
echo $waiting_job_id
curl --request POST \
--url https://circleci.com/api/v2/workflow/$workflow_id/approve/$waiting_job_id \
--header "Circle-Token: $CIRCLECI_API_KEY" \
--header "content-type: application/json"
fi
when: always
...
The script first checks for the existence of the triggering-pipeline-id pipeline parameter. It proceeds only if that parameter exists. The when: always line in the command makes sure that this executes regardless of termination status.
Then it makes 3 API calls:
- Get workflow ID in a pipeline. There is just one workflow in that pipeline for this sample project.
- Get jobs in that workflow, use
jqto select the one that matches the approval job name (wait-for-triggered-pipeline), and extract the approval job’s ID. - Make the request to the approval endpoint with the waiting job ID.
For this tutorial, we are storing results like workflow ID and job ID in local bash variables, and using them in subsequent calls to the API.
Note: If you have more jobs than can be sent in a single response, you might have to handle pagination as well.
Now that you have made the approval request, pipeline B is complete, and pipeline A should be running again.
Update pipeline A with the result of B
After pipeline A has been approved, the next job in the workflow will begin. If your workflow graph requires it, pipeline A can trigger multiple jobs.
We still do not have any indication of the result of the previous workflow. To get that information, you can use the API again to get B’s status from pipeline A. An example job could look like that: check-status-of-triggered-pipeline.
First, you need to retrieve the ID of the triggered pipeline, which is pipeline B. This is the same ID that was persisted in a workspace in an earlier step. Retrieve it using cat:
- attach_workspace:
at: workspace
- run:
name: Check triggered workflow status
command: |
triggered_pipeline_id=$(cat workspace/pipeline.txt)
Then use the API to retrieve the workflow. Use jq to get just the status of the first item in the returned array of workflows:
created_workflow_status=$(curl --request GET \
--url "https://circleci.com/api/v2/pipeline/${triggered_pipeline_id}/workflow" \
--header "Circle-Token: $CIRCLECI_API_KEY" \
--header "content-type: application/json" \
| jq -r '.items[0].status'
)
Check that status is not success. If it is not, use exit to terminate the job with exit code -1. If the workflow is successful, it will terminate:
if [[ "$created_workflow_status" != "success" ]]; then
echo "Workflow not successful - ${created_workflow_status}"
(exit -1)
fi
echo "Created workflow successful"
Here is the full config for the job check-status-of-triggered-pipeline:
check-status-of-triggered-pipeline:
docker:
- image: cimg/base:2021.11
resource_class: small
steps:
- attach_workspace:
at: workspace
- run:
name: Check triggered workflow status
command: |
triggered_pipeline_id=$(cat workspace/pipeline.txt)
created_workflow_status=$(curl --request GET \
--url "https://circleci.com/api/v2/pipeline/${triggered_pipeline_id}/workflow" \
--header "Circle-Token: $CIRCLECI_API_KEY" \
--header "content-type: application/json" \
| jq -r '.items[0].status'
)
echo $created_workflow_status
if [[ "$created_workflow_status" != "success" ]]; then
echo "Workflow not successful - ${created_workflow_status}"
(exit -1)
fi
echo "Created workflow successful"
Conclusion
In this article we have reviewed an example of a complex pipeline orchestration pattern that I have named “circleback”. The circleback pattern creates a dependent pipeline and allows you to wait for it to terminate before completing. It involves making several API keys from both projects, the use of an approval job, and the workspace feature of CircleCI to store and pass values such as pipeline ID across jobs in a workflow. The sample projects are located in separate repositories: project A, and project B.
If this article has helped you in a way, I would love to know, also if you have any questions or suggestions about it, or ideas for future articles and guides, reach out to me on Twitter - @zmarkan or email me.
