Conditional CircleCI pipeline execution

The DevOps practice of continuous integration and continuous deployment (CI/CD) improves software delivery. CI/CD platforms monitor and automate the application development process ensuring a better application, faster. CI/CD pipelines build code, run tests, and deploy a production-ready version of an application that has passed all automated checks.

With the ongoing need to ensure process integrity, development teams may want to execute pipeline jobs based on the results of the preceding job. This tutorial will teach you about conditional pipelines and how to optionally execute jobs in a CI/CD workflow depending on conditions you specify. I will explain how to set up pipelines, how to set conditions for their execution, and how failure of job A affects the execution of job B and contributes to the results of the pipeline.

Prerequisites

The following are required to complete this tutorial:

• Node.js installed on your system
• A CircleCI account
• A GitHub account
• An understanding of how CircleCI pipelines work

With these prerequisites in place, you are ready to start the tutorial.

Understanding conditional pipeline execution

Most pipelines are made up of multiple steps that must be completed before the final version of the software is delivered. You can skip one or more pipeline stages based on a condition or set of conditions you specify.

To be effective, a pipeline should have clearly defined steps that must be verified before proceeding to the next step. Here is an example of a pipeline with four jobs:

• lint check
• code build
• testing
• deployment

Each job is dependent on the previous one being verified, or running successfully. If you want to deploy an application that has passed all of its tests, you have created a condition for your CI/CD pipeline. That is, the code and build steps must pass before the deployment step can be executed.

To give you a better understanding of conditional pipeline execution, I have created an illustration of the three-step pipeline.

There are three pipelines, each with three jobs. In the first pipeline, all jobs are verified, and the pipeline succeeds. However, in the second pipeline, the first job passes, triggering build and test, but a failure in the second step then blocks the deployment job. The pipeline fails.

Pipeline 3 differs from the others in that if the first job fails, the process is immediately halted.

Setting up an API to run lint before tests execute

This section of the tutorial will demonstrate how failing lint leads to pipeline failure.

Making sure that your code has proper formatting and linting in place provides a good foundation for running tests and builds. Failure of one job, (if your code is not properly linted for example) should prevent triggering of the build and test phases. The pipeline will not meet the conditions and will fail, which is as expected.

I have built an API for you to use with this tutorial. All you need to do is clone the repository:

git clone https://github.com/CIRCLECI-GWP/conditional-pipeline-execution-demo.git

This command clones the project into your local machine into a directory called conditional-pipeline-execution-demo. This should be in your current working directory where you cloned the repository. Once this is done, run the commands below.

Note: To run the pipelines on CircleCI, I recommend that you create a Mongo Atlas account and set it up according to the instructions provided in the README.md of the cloned repository.

Change directory:

cd  conditional-pipeline-execution-demo

Install the necessary dependencies:

npm install

After the installation is complete, use this command to start the server:

npm  start

The API for this tutorial is now running on your local machine. You can test it by visiting http://localhost:3000/.

Good news: this API is already properly configured with Prettier and ESLint. The lint and Prettier configurations are handled in the package.json file. Some tests have also been written.

Next, make sure that the repository formatting follows the configured Prettier guidelines. Run the following commands to start the lint and formatting checks:

npm run format:check # checks if formatting guidelines are followed

Format all files with Prettier:

npm run format:write # Writes onto formatted files applying the eslint and prettier rules

Lint code:

npm run lint:check # checks if code is linted

Now that you have verified that format and linting are working as expected, you can configure the CircleCI pipelines, including optionally executing the pipelines when conditions are met.

CircleCI configuration file setup

Make a .circleci directory in the root directory of your API. Inside it, create an empty file called config.yml and add this:

# CircleCI configuration file

version: 2.1
workflows:
   lint-build-test:
       jobs:
           - lint-code
           - build-and-test:
                 requires:
                     - lint-code
jobs:
   lint-code:
       docker:
           - image: cimg/node:16.13.2
       steps:
           - checkout
           - run:
                name: update npm
                command: 'sudo npm install -g npm'

           - restore_cache:
                key: dependency-cache-{{ checksum "package-lock.json" }}

           - run:
                name: install dependencies
                command: sudo npm install

           - save_cache:
                key: dependency-cache-{{ checksum "package-lock.json" }}
                paths:
                    - ./node_modules

           - run:
                 name: lint check
                 command: npm run format:check

    build-and-test:
        docker:
            - image: cimg/node:16.13.2
        steps:
            - checkout
            - run:
                    name: update npm
                    command: 'sudo npm install -g npm'

            - restore_cache:
                    key: dependency-cache-{{ checksum "package-lock.json" }}

            - run:
                    name: install dependencies
                    command: sudo npm install

            - save_cache:
                key: dependency-cache-{{ checksum "package-lock.json" }}
                paths:
                        - ./node_modules

            - run:
                    name: run tests
                    command: npm test

This config.yml file has a workflow with two jobs: lint-code and build-and-test.

Note the use of the requires key in the workflow definition specifying that lint-code is required for the build-and-test job to execute successfully. This means that the second job will be executed only when the first one succeeds. When the first job fails, the process is automatically halted, so a condition is set here. Lint checks must pass for our pipeline to build and test the code successfully.

Following the workflow, we will define the actual jobs. The first step in both jobs is to instruct CircleCI to use the node docker image. Then they will update the node package manager and use npm to install the required dependencies. The last step in both jobs is to carry out these three actions: lint, check, and test. The configuration also uses the restore_cache and save_cache steps to cache the dependencies and then restore them. These steps significantly reduce the time it takes to execute the pipeline.

Pushing code to GitHub

Note: If you cloned the repository the changes already exist in the repository, making this an optional step. If you are using a different repository and using the same configuration, you will need to push the changes to that repository.

Save your work, commit, and push changes to the GitHub repository. From the CircleCI dashboard Projects section, click Set up Project beside the repository name.

When you are prompted, use main, which is our default branch. Then click Set Up Project start running the project on CircleCI.

The CircleCI dashboard will show that this build failed, which is as expected. This perfectly introduces us to the next section of this tutorial about conditional execution of pipelines.

Conditional execution of pipelines

Our build failed because of lint issues that we intentionally introduced. For the pipeline to be effective, you need to be able to stop it when a set of conditions are not met. You should be able to terminate deployment if tests fails, or in this case when linting of the project fails.

• The lint check step in the pipeline shows how a failure in one step cascades to the rest of the pipeline. CircleCI skipped running our tests because it encountered a failure when executing the command npm run format:check from our package.json file in the lint-code job. Successful completion of that job is a requirement for the build-test job.
• The format:check command runs the prettier --list-different . command, which verifies that all the files use the Prettier configuration. If it finds files that don’t match the configuration, it throws an exception. When unlinted files are encountered, the exception is thrown on CI and the pipeline fails.

The intentional failure of the pipeline, due to failed lint, means you can now verify that if a step fails when the run is not a parallel run, the subsequent steps will not execute and the pipeline will fail. In this case we have declared this explicitly in our configuration in this code snippet:

# congig.yml line 6, 7, and 8
- build-and-test:
    requires:
      - lint-code

There you have it - you have configured a pipeline to run specific jobs only when a certain condition is met. You can also write parallel pipelines, each with a different set of conditions. CircleCI can easily interpret this and properly execute your pipeline.

Advanced conditional execution

In this section, I will provide more detail about setting up pipelines to run or not run based on conditions. This section covers:

• Holding a workflow for a manual approval
• Using “when” and “or” logic statements

Holding a workflow for a manual approval

We can configure workflows to require manual job approval before moving on to the next job. Anyone with push privileges to the repo can approve a workflow on hold by clicking the Approval button, which resumes the workflow.

We already have a workflow set up. You can modify it by adding a job with the key type: approval to the jobs list.

workflows:
  lint-build-test:
    jobs:
      - lint-code
      - hold:
          type: approval
          requires:
            - lint-code

      - build-and-test:
          requires:
            - hold

This code adds a new job to the workflow named hold. The type: approval is the key-value pair that will put the workflow on hold. In the real world, you can hold a pipeline for purposes such as approval from QA or a deployment manager or to manually prevent accidental deploys. Using the requires key sets a condition in the hold job to run it when lint-code succeeds. After the hold job is approved, any subsequent jobs that require it will run. A user can then manually continue the build and test job.

Commit these changes to GitHub. On the CircleCI dashboard, the first job executes, then holds and waits for approval.

Click the On Hold tab from the dashboard to go to the next step, which includes a visual representation of the workflow. Click the hold box and then approve the job in the pop-up window.

Once the job is approved, the workflow resumes, and the subsequent jobs can run.

Using logic statements to control pipeline execution

In addition to using the requires keyword to define workflow conditions, you can also use the when clause in conjunction with a logic statement. When logic statements are used in a when clause, they resolve to a true or false value. In this example, you will modify your previous workflow and execute it only when you modify either a staging branch or a main branch. Make the changes shown in this code block:

workflows:
  lint-build-test:
    when:
      or:
        - equal: [main, << pipeline.git.branch >>]
        - equal: [staging, << pipeline.git.branch >>]
    jobs:
      - lint-code
      - hold:
          type: approval
          requires:
            - lint-code
      - build-and-test:
          requires:
            - hold

The when and or clause sets a condition that prevents the workflow lint_build_test from running unless the workflow is on the branches staging or main.

Now, if you run this on any other branch, the workflow will fail to execute.

With these flags, you can provide rules like “releases can only be made on the release branch” or “deployments must take place only from the main branch” to avoid accidental pre-production deploys.

Verifying successful build

Now that you know how to write conditional pipelines and how the conditions affect pipeline outcomes, it is time to fix the linting issues and push the code to GitHub. When the change is made, your pipeline executes successfully and both the lint-code and build-and-test jobs are executed successfully.

Voila! Your pipeline is now green. It runs the lint-check pipeline and then proceeds to build and test the code. In this celebratory mode, it is time to wrap up this tutorial.

Conclusion

In this tutorial, you learned about conditional pipelines and how to build them. You executed a job in a CircleCI pipeline based on the condition that the first job must pass before the second one can be executed. You even tried out a couple of more advanced concepts: holding a workflow and using when and or statements. You have first-hand experience with the failure of one job causing a cascading failure that shuts down the entire pipeline. You have also learned how to enforce controls through in-built CircleCI features. And you were able to observe your conditional pipelines executing successfully. I hope you enjoyed this tutorial and found it useful. Until next time, keep learning!