Adding application and image scans to your CI/CD pipeline

Have you ever heard someone say, “security is everyone’s job”? I’ve been hearing this now for the majority of my career. Like most people that have had the “honor” of hearing this phrase, primarily spoken by security team members, I, too, rolled my eyes after it graced my ears. I remember being offended and thinking, “what gives this person the right to shed their duties onto me?” It wasn’t until later in my career, after experiencing a couple of security incidents, that I realized what the phrase meant. They were correct in saying that security is everyone’s job.

DevSecOps

DevSecOps is the philosophy of weaving security practices into the software development life cycle (SDLC). This means adding security checks during the application’s development and not as an afterthought. Building security into DevOps practices is not a new concept, but in my experience, security practices have generally been tacked on at the end of the SDLC.

You may be asking, “What difference does it make where and when security operations are performed, as long as they are getting done?” Well, here is a scenario that might clear things up. Imagine that you just developed some awesome, new features for a project that you’re working on. You’ve programmed the new features, written all the corresponding tests, and your work is passing on your CI/CD platform. You’re excited to get your work deployed, but at the end of this process, your release gets flagged by security. They manually performed a scan of your software and the scan reported high-risk vulnerabilities in some of the dependency libraries being used in the project. You now have to remediate all of the security issues and then run through the whole process all over again.

DevOps enables us to deliver quality software rapidly and efficiently. However, the scenario I mentioned above is not efficient. Automating and implementing security processes into CI/CD pipelines rectifies this. By integrating security scans into segments of CI/CD pipelines, teams will be alerted to security discrepancies very early on in the development process. Fixing vulnerable libraries is easier to mitigate the earlier they are surfaced. Consider the possibility that the updated libraries have deprecated some features in the vulnerable version which were heavily used in the app. This is a huge problem and it will require some re-engineering to accommodate the changes in the updated libraries, tacking on unexpected time and effort to the development process. Again, integrating security into CI/CD pipelines will alert you to security issues early on and will enable you to fix them sooner in the process rather than later.

Now that I’ve shared what DevSecOps is and some of its benefits, I’d like to demonstrate how to easily add security scans to your CI/CD pipelines using a security scan tool called Snyk.

Snyk

CircleCI partners with a host of security companies, but for the purpose of this post, we will be using Snyk to integrate security checks. Snyk enables you to find, and more importantly, fix known vulnerabilities in your applications and containers. Snyk is a CircleCI technology partner and they created the Snyk orb which enables developers to add security scans to their CI/CD pipelines. In this post, I’ll demonstrate how easy it is to add security to your pipelines.

Prerequisites

Before you begin, you need a few things in place:

• A CircleCI account
• A Snyk account
• A Docker Hub account
• A project/repo in CircleCI
• Add these environment variables to the CircleCI project via the dashboard:
  • $SNYK_TOKEN
  • $DOCKER_LOGIN - Your Docker Hub username
  • $DOCKER_PWD - Your Docker Hub password

Once all of these prerequisites are completed, we can move onto the next section.

Base pipeline configuration with no security

Let’s start with a base pipeline configuration so that you can see a pipeline without a security integration. I’ve been working a lot with Python applications these days, so below is an example pipeline configuration for a Python app that doesn’t include any security actions.

version: 2.1
jobs:
  build_test:
    docker:
      - image: circleci/python:3.7.4
    steps:
      - checkout
      - run:
          name: Install Python Dependencies
          command: |
            echo 'export PATH=~$PATH:~/.local/bin' >> $BASH_ENV && source $BASH_ENV
            pip install --user -r requirements.txt
      - run:
          name: Run Unit Tests
          command: |
            pytest
  build_push_image:
    docker:
      - image: circleci/python:3.7.4
    steps:
      - checkout
      - setup_remote_docker:
          docker_layer_caching: false
      - run:
          name: Build and Push Docker image to Docker Hub
          command: |
            echo 'export PATH=~$PATH:~/.local/bin' >> $BASH_ENV
            echo 'export TAG=${CIRCLE_SHA1}' >> $BASH_ENV
            echo 'export IMAGE_NAME=orb-snyk' >> $BASH_ENV && source $BASH_ENV
            pip install --user -r requirements.txt
            pyinstaller -F hello_world.py
            docker build -t $DOCKER_LOGIN/$IMAGE_NAME -t $DOCKER_LOGIN/$IMAGE_NAME:$TAG .
            echo $DOCKER_PWD | docker login -u $DOCKER_LOGIN --password-stdin
            docker push $DOCKER_LOGIN/$IMAGE_NAME
workflows:
  build_test_deploy:
    jobs:
      - build_test
      - build_push_image:
          requires:
            - build_test

This pipeline configuration accomplishes the following:

• Installs application dependencies defined in the requirements.txt manifest file
• Executes the application’s unit tests
• Creates/builds a new Docker image for the application
• Publishes the newly created Docker image to the Docker Hub registry for later use

The goal of this pipeline is to build, test, and deploy the code via a Docker image. At no point in this pipeline are there any security or vulnerability scans.

Security enabled pipeline - the Snyk app scan

Now you’ve seen a glimpse of an “insecure” pipeline, and it should make your skin crawl. Next, I’m going to show you an example of a security-enabled pipeline configuration.

version: 2.1
orbs:
  snyk: snyk/snyk@0.0.8
jobs:
  build_test:
    docker:
      - image: circleci/python:3.7.4
    steps:
      - checkout
      - run:
          name: Install Python Dependencies
          command: |
            echo 'export PATH=~$PATH:~/.local/bin' >> $BASH_ENV && source $BASH_ENV
            pip install --user -r requirements.txt
      - snyk/scan
      - run:
          name: Run Unit Tests
          command: |
            pytest
  build_push_image:
    docker:
      - image: circleci/python:3.7.4
    steps:
      - checkout
      - setup_remote_docker:
          docker_layer_caching: false
      - run:
          name: Build and Push Docker image to Docker Hub
          command: |
            echo 'export PATH=~$PATH:~/.local/bin' >> $BASH_ENV
            echo 'export TAG=${CIRCLE_SHA1}' >> $BASH_ENV
            echo 'export IMAGE_NAME=orb-snyk' >> $BASH_ENV && source $BASH_ENV
            pip install --user -r requirements.txt
            pyinstaller -F hello_world.py
            docker build -t $DOCKER_LOGIN/$IMAGE_NAME -t $DOCKER_LOGIN/$IMAGE_NAME:$TAG .
            echo $DOCKER_PWD | docker login -u $DOCKER_LOGIN --password-stdin
            docker push $DOCKER_LOGIN/$IMAGE_NAME
workflows:
  build_test_deploy:
    jobs:
      - build_test
      - build_push_image:
          requires:
            - build_test

Before I explain what’s going on in this pipeline, I want to mention that Snyk has many capabilities, but for the purposes of this post, I’m only going to cover the application and Docker image scans. The above example demonstrates the application scan. Now, I’ll explain the new parts that I included to make the pipeline more secure.

The pipeline block below uses the Snyk orb to easily integrate the Snyk tool into the pipeline. This block is equivalent to an import or include statement in a scripting or programming language. In this block, you’re also declaring the version of the Snyk orb you’d like to use.

version: 2.1
orbs:
  snyk: snyk/snyk@0.0.8

The next pipeline block defines the Docker image used to run the build. It then does a checkout or “git clone” of your source code into the container. Following that, the run: block will install the dependencies listed in the requirements.txt file. This file lists all of your application libraries and dependencies which can be considered a Software Bill of Materials (SBOM) specific to the Python aspects of the project. It also feeds the list of software to Snyk, so that it knows what to scan and test.

jobs:
  build_test:
    docker:
      - image: circleci/python:3.7.4
    steps:
      - checkout
      - run:
          name: Install Python Dependencies
          command: |
            echo 'export PATH=~$PATH:~/.local/bin' >> $BASH_ENV && source $BASH_ENV
            pip install --user -r requirements.txt

Below you’ll find the contents of the requirements.txt file that was previously discussed.

The next block is where we execute some DevSecOps action within the pipeline. - snyk/scan calls the scan command from the Snyk orb. It will read the requirements.txt file, and then compare that list of software against the Snyk vulnerability databases to look for any matches. If there are any matches, Snyk will flag it and fail this segment of the pipeline. The goal here is to alert teams to security issues as early as possible so that they can be quickly mitigated and the CI/CD process can securely continue.

      - snyk/scan
      - run:
          name: Run Unit Tests
          command: |
            pytest

The remainder of the example pipeline configuration deals with the Docker image build segments.

After the Snyk application security scan is complete, your build will pass if there are no vulnerabilities detected. If the scan detects a vulnerability, the build will fail. This is the Snyk orb’s default behavior (see the Snyk orb parameters for more details on these parameters). Along with the pipeline failing after the scan, Snyk will provide a detailed report on why the build failed.

Below is an example of a failed application security scan reported in the CircleCI dashboard.

The scan results provide useful details regarding the security scan. They show that the application is using a vulnerable version of the Flask library, 0.12.4. The results also suggest a fix. The fix requires upgrading Flask to the newer 1.0 version.

You could also see results for this failed scan from the Snyk dashboard. That dashboard has an even greater level of detail. The image below shows an example of the Snyk dashboard.

The security scan provides insight into the application’s vulnerabilities and useful suggestions on how to mitigate the issue so that teams can fix them and move on to the next task. Some fixes are more involved than others, but being alerted to the issues is a very powerful feature.

Security enabled pipeline - the Snyk Docker image scan

I’ve discussed the Snyk application scanning capabilities above and now I want to discuss the Docker image scanning capabilities that are also easily integrated into CI/CD pipelines. The build_push_image: block shown below is from the previous Base pipeline configuration with no security example.

  build_push_image:
    docker:
      - image: circleci/python:3.7.4
    steps:
      - checkout
      - setup_remote_docker:
          docker_layer_caching: false
      - run:
          name: Build and Push Docker image to Docker Hub
          command: |
            echo 'export PATH=~$PATH:~/.local/bin' >> $BASH_ENV
            echo 'export TAG=${CIRCLE_SHA1}' >> $BASH_ENV
            echo 'export IMAGE_NAME=orb-snyk' >> $BASH_ENV && source $BASH_ENV
            pip install --user -r requirements.txt
            pyinstaller -F hello_world.py
            docker build -t $DOCKER_LOGIN/$IMAGE_NAME -t $DOCKER_LOGIN/$IMAGE_NAME:$TAG .
            echo $DOCKER_PWD | docker login -u $DOCKER_LOGIN --password-stdin
            docker push $DOCKER_LOGIN/$IMAGE_NAME

The run block above has no security scans integrated. This increases the risk of exposure to severe vulnerabilities within the Docker image. Synk’s Docker image scan is an essential feature for checking your Docker images for vulnerabilities. Below is an example of how to integrate the Snyk Docker image scans into your pipeline configurations.

  build_push_image:
    docker:
      - image: circleci/python:3.7.4
    steps:
      - checkout
      - setup_remote_docker:
          docker_layer_caching: false
      - run:
          name: Build and Scan Docker image
          command: |
            echo 'export PATH=~$PATH:~/.local/bin' >> $BASH_ENV
            echo 'export TAG=${CIRCLE_SHA1}' >> $BASH_ENV
            echo 'export IMAGE_NAME=orb-snyk' >> $BASH_ENV && source $BASH_ENV
            pip install --user -r requirements.txt
            pyinstaller -F hello_world.py
            docker build -t $DOCKER_LOGIN/$IMAGE_NAME -t $DOCKER_LOGIN/$IMAGE_NAME:$TAG .
      - snyk/scan:
          fail-on-issues: true
          monitor-on-build: true
          docker-image-name: $DOCKER_LOGIN/$IMAGE_NAME:$TAG
          target-file: "Dockerfile"
          project: ${CIRCLE_PROJECT_REPONAME}/${CIRCLE_BRANCH}-app
          organization: ${SNYK_CICD_ORGANIZATION}
      - run:
          name: Push Docker image to Docker Hub
          command: |
            echo $DOCKER_PWD | docker login -u $DOCKER_LOGIN --password-stdin
            docker push $DOCKER_LOGIN/$IMAGE_NAME

The pipeline configuration block example above is significantly different than the original and I’ll discuss those differences now.

The block below is a new - run: block that sets up some environment variables that are used to name and version the Docker image being built. The last line in this block builds the image.

      - run:
          name: Build and Scan Docker image
          command: |
            echo 'export PATH=~$PATH:~/.local/bin' >> $BASH_ENV
            echo 'export TAG=${CIRCLE_SHA1}' >> $BASH_ENV
            echo 'export IMAGE_NAME=orb-snyk' >> $BASH_ENV && source $BASH_ENV
            pip install --user -r requirements.txt
            pyinstaller -F hello_world.py
            docker build -t $DOCKER_LOGIN/$IMAGE_NAME -t $DOCKER_LOGIN/$IMAGE_NAME:$TAG .

The next block below is where the Docker image security scan is declared and executed. - snyk/scan: is the same command used in the previous app security scan with some differences. In order to execute the Docker image scans from the - snyk/scan: command, you have to declare and set values for the docker-image-name: and target-file: parameters. Again, I suggest that you familiarize yourself with the Snyk orb parameters to understand the tool’s capabilities.

      - snyk/scan:
          fail-on-issues: true
          monitor-on-build: true
          docker-image-name: $DOCKER_LOGIN/$IMAGE_NAME:$TAG
          target-file: "Dockerfile"
          project: ${CIRCLE_PROJECT_REPONAME}/${CIRCLE_BRANCH}-app
          organization: ${SNYK_CICD_ORGANIZATION}

Below is an example of a failed Docker scan within a pipeline as reported in the CircleCI dashboard. The details shown appear similar to the app scan but are definitely different. It shows that vulnerabilities in the Docker image stem from the base image which is python:3.7.4 This image is published by the language maintainers.

Like the Snyk app scan, more details regarding the failed Docker image scan can be found from the Snyk dashboard. The Snyk dashboard will have all the details needed to mitigate the detected vulnerabilities. An example of a failed Docker image scan from the Snyk dashboard is shown below.

![Example of a Docker image scan failure within the snyk dashboard.]

The last pipeline block accesses Docker Hub and pushes and publishes the scanned Docker image to the Docker Hub registry. This is the last command executed in the build_push_image: block.

      - run:
          name: Push Docker image to Docker Hub
          command: |
            echo $DOCKER_PWD | docker login -u $DOCKER_LOGIN --password-stdin
            docker push $DOCKER_LOGIN/$IMAGE_NAME

Wrapping up

In this post, I’ve discussed DevSecOps and the benefits that adoption and integration into CI/CD pipelines provides. Teams that build applications with security integrated into the development cycle early on save time and effort and avoid potential security incidents. This means integrating security scans into every possible segment of the CI/CD pipeline to ensure that your application and infrastructure are aptly protected.

Using tooling from CircleCI partners such as Snyk’s orb will enable teams to easily integrate DevSecOps practices into their CI/CD pipelines, shielding their applications from known threats and vulnerabilities.

Do you have feedback regarding this post? Feel free to @ me on twitter @punkdata to start a conversation about DevSecOps.

Thanks for reading!