Modern Continuous Integration/Continuous Deployment (CI/CD) pipelines are designed to consume, manipulate, and update data from external services/platforms. An example of this would be an application test that requires data from a Customer Relationship Management (CRM) service to verify that the data output formats are valid for consumption by the application. Since the CRM is a service hosted externally, the CI/CD job will need a mechanism to access the data. REST API calls are excellent for these types of situations.

In this blog post, we will discuss REST APIs and some of the reasons for integrating them into your CI/CD pipelines.

API (Application Programming Interface)

APIs are a set of clearly defined methods for communication between components within an application or application service. They provide a mechanism for developers to programmatically interface with other code/modules and engage their functions. A mobile application may need to update data, delete data, or query customer data in its database. The application uses an API’s functions to communicate the actions from the app to the database. APIs are comparable to Graphical User Interfaces. They both enable users with functionality while doing the majority of the work behind the scenes. APIs, though, are only accessible via code.

REST APIs (Representational State Transfer)

REST or RESTful APIs are currently the most popular APIs. They can be used in almost any protocol and they offer a great deal of flexibility. REST APIs are stateless, meaning that calls can be made independently of one another. Each call contains all of the necessary data needed to complete itself successfully. The focus of this post will be REST APIs.

CI/CD and REST APIs

Organizations are increasingly adopting distributed computing and cloud-based technologies and implementing them into their services and offerings. Distributed systems rely heavily on REST APIs for cross-platform/service communications and interaction. The system’s dependence on REST APIs will naturally cascade into their CI/CD pipelines in the form of testing or deployments. Our earlier example of the application test shows us how the testing phase of the CI process can require access to data from an external data source as well as extensive validation of data and application behavior.

REST APIs can be critical in CD processes, especially in the case of deploying builds to various application environments that are hosted on cloud-based infrastructures. The CD process can use REST API calls to interact with the host in order to initiate and monitor an application deployment to a target environment.

In both the CI and CD processes described above, REST APIs are key in providing the access and interactions required to successfully complete the CI/CD pipeline.

Accessing REST APIs

REST APIs are generally secured via authentication & authorization mechanisms such as Basic Auth and OAuth. These authentication methods ensure that the REST API services and their data are protected from unauthorized access. Make sure that the REST APIs you are using have a valid HTTPS certificate. As a best practice, always access REST APIs using HTTPS/TLS encryption. This protects against transmitting information in clear text over networks that can be easily retrieved and read by unauthorized parties. Most modern REST API client tools and libraries will flag and notify you of transactions that are performed over unencrypted protocols and some will not even function over them.

Your method of securely accessing APIs will vary from provider to provider. Some enable the usage of static API keys and secrets. Others require authentication via refresh and access tokens that are generated and distributed in an OAuth ceremony. To find out what method to use, read through the documentation of the REST API providers.

Integrating REST APIs into CI/CD

Integrating REST APIs into CI/CD pipelines is accomplished in many ways depending on the API providers access method. Some providers enable access to their services using command line interface (CLI) tools such as:

Wget, cURL, and similar tools enable direct calls to any REST API and require deep knowledge of the target API. These tools give users lots of flexibility and unfettered control over the requests made to the targeted api. Other types of CLI tools are designed to access APIs in a more restrictive manner. This is intended to ease user interaction and standardize request results. CLI tools give users a nice clean way of interacting with specific REST APIs and are generally offered and maintained by the API service provider. For example, AWS offers and maintains their AWS CLI Tool which is widely used by developers, system administrators, and DevOps teams to interact with the AWS cloud platform.

Below is an example of a CircleCI config.yml that shows you how to use a cURL request against the GitHub REST API call to provide a real-time status of the GitHub service:

version: 2
jobs:
  build:
    docker:
      - image: circleci/python:2.7.14
        environment:
          FLASK_CONFIG: testing
    steps:
      - run:
          name: Get GitHub API Status
          command: |
            GH_URL='https://status.github.com/api/status.json'
            GH_STATUS=$(curl -s $GH_URL |  jq --raw-output '.status')
            echo "GitHub API Status: $GH_STATUS"
            if [ "$GH_STATUS" != "good" ]; then
              echo "Error [GitHub API status: $GH_STATUS]"
              exit 1
            fi    
      - checkout
      - run:
          name: Setup VirtualEnv
          command: |
            echo 'export TAG=0.1.${CIRCLE_BUILD_NUM}' >> $BASH_ENV
            echo 'export IMAGE_NAME=cicd-101-workshop' >> $BASH_ENV 
            virtualenv helloworld
            . helloworld/bin/activate
            pip install --no-cache-dir -r requirements.txt
      - run:
          name: Run Tests
          command: |
            . helloworld/bin/activate
            python test_hello_world.py
      - setup_remote_docker:
          docker_layer_caching: true
      - run:
          name: Build and push Docker image
          command: |
            . helloworld/bin/activate
            pyinstaller -F hello_world.py
            docker build -t $DOCKER_LOGIN/$IMAGE_NAME:$TAG .
            echo $DOCKER_PWD | docker login -u $DOCKER_LOGIN --password-stdin
            docker push $DOCKER_LOGIN/$IMAGE_NAME:$TAG

The run block named Get GitHub API Status gets status of the API and verifies that the GitHub service is accessible and operating nominally. It uses the cURL client and a bash if statement. If the GitHub API status result is anything other than good, the build will fail and exit. This will prevent the build from continuing if the GitHub service is unavailable, saving time and money.

- run:
    name: Get GitHub API Status
    command: |
      GH_URL='https://status.github.com/api/status.json'
      GH_STATUS=$(curl -s $GH_URL |  jq --raw-output '.status')
      echo "GitHub API Status: $GH_STATUS"
      if [ "$GH_STATUS" != "good" ]; then
        echo "Error [GitHub API status: $GH_STATUS]"
        exit 1
      fi

Summary

In this post, we discussed APIs and looked at an example of integrating cURL into a CI/CD pipeline. Many pipelines require access to external data or functions that trigger actions on related platforms. REST API integrations provide this functionality. In the modern world of distributed computing, REST APIs are a staple in platform and data interchange.

If you want to learn more about CircleCI check out the documentation site. If you get stuck, you can reach out to the CircleCI community via the community/forum site.