Welcome to the DevSecOps and CI/CD security guide. Browse through each section to discover various relevant resources to ensure security of your applications and infrastructure.
What is DevSecOps?
DevSecOps is the philosophy of developing applications and infrastructure securely from ideation to deployment. It requires consideration of security risks at all stages of the development lifecycle. While DevOps teams have historically focused on automating the building, testing, and deployment of their applications, DevSecOps includes automating security practices to allow teams to increase security without losing velocity.
Secure CI/CD pipelines with DevSecOps
Secure CI/CD pipelines are part of daily operations for many businesses. These processes, when set up correctly, keep the delivery process consistent by automating many manual tasks and showing how the software is being worked on.
CI/CD is also the place in your technology stack where your infrastructure has access to many different resources, from development and production environment to analytics keys and code signing credentials. The more resources your pipelines have access to (secure secrets, proprietary code, and databases), the more important it is to keep your CI/CD system secure.
In this article, we talk about the different parts of security for CI/CD pipelines and highlight a number of ways to improve on these aspects.
Three categories of security best practices for CI/CD
We usually divide security practices into three parts that you can address with different kinds of solutions. These are:
- Secure pipeline configuration
- Code and Git history analysis
- Security policy enforcement
All three aspects are equally important. Let’s take a deeper look at these categories and ways to address them.
Secure pipeline configuration
It is possible to use your CI/CD pipeline configuration to make security issues less likely to happen.
First, safely store the secrets in your pipelines that connect to databases and third-party services. On CircleCI, you can use encrypted-at-rest environment variables, or the contexts feature. Contexts provide access to environment variables across projects. Their use can also be restricted to specific security group members as defined by the organization’s administrator.
Another option is to use a third-party solution to dynamically fetch secrets from their secure storage for your jobs. To ensure that none of your credentials become exposed to those who shouldn’t see them, do not check these into your repository in plain text, even if the repository is private.
Second, for sensitive files like code signing keys we recommend adding an additional layer of isolation between your encrypted files and your repository. See an example here. Keep them encrypted in your repository and store the decryption key in the environment variables or contexts. Decrypt them inside your CI/CD jobs only when they are needed. This makes it even more difficult to expose or leak these highly sensitive files.
Third, never leave your CI/CD environment running without monitoring it. Make sure that the containers and VMs used for your pipelines are destroyed once jobs that require sensitive information are complete. On CircleCI, this happens automatically for all supported platforms including Linux on Docker and Machine, macOS, and Windows.
Fourth, pay extra attention to how your CI/CD system handles builds for pull requests that come from forks of your repository. If your build stages require secrets to work, forked pull requests might gain access to them through your pipelines. On CircleCI, the default behavior is to never pass on the secrets to the forked pull requests.
By addressing these four aspects you can significantly reduce the risk of security incidents due to your CI/CD pipeline configuration. Next up, code and Git history analysis.
Code and Git history analysis
We all love Git for its comprehensive history of changes, being able to look through the history of the project with a simple CLI command and see who has most recently changed a line of code, among other benefits. This also means that any sensitive information that makes it into the Git history can be accessible to attackers if a Git repository is exposed, even if the most recent state of the repository no longer contains the secrets.
Trufflehog and GitLeaks are tools that help you identify secrets that have been committed to the codebase so that you can deactivate and replace them. These tools scan your Git history for traces of secrets that your team might have added to the repository in the past.
Once your Git history is clear of secrets, you can proceed to the next level: making sure that your current revision doesn’t contain any vulnerable dependencies.
Static Application Security Testing (SAST) techniques can look through the application in your commit and analyze its dependencies. If dependencies contain any issues or known security vulnerabilities, your commit will be marked as insecure and won’t proceed to deployment.
Dynamic Application Security Testing (DAST) techniques go one step further and spin up a copy of your production environment inside your CI job so you can scan the containers and executables. The dynamic aspect helps the system catch dependencies that are being loaded at launch time, for example, because those won’t be caught by SAST.
Security policy enforcement
Some security aspects can’t be statically checked based on known vulnerabilities, but rather are specific to your particular company. These need to be codified as policies, which can be automated or manual compliance checks.
Certain tasks like reviewing the list of all accounts that have access to your repositories, or making sure your onboarding and offboarding processes are in sync, will be manual. We recommend that you set reminders to perform these regularly.
Some tasks actually can be automated quite easily. For example, third-party services can provide a convenient way to codify a set of rules to match against your CI pipeline. These services can help you prove compliance with the regulations that govern your data. If there is a mismatch, the build will fail.
Securing your CI/CD pipelines with CircleCI orbs
CircleCI orbs are shareable packages of CircleCI configuration to use in your builds. Orbs define reusable commands, executors, and jobs so that commonly used pieces of configuration can be condensed into a single line of code. Orbs are reusable, shareable, open source packages of CircleCI config that enable the immediate integration of these services. With orbs, you get an out-of-the-box solution for securing your pipeline.
Looking for orb-specific resources? Check out the following blog posts:
- DevSecOps and CircleCI orbs: secure your CI/CD pipeline
- Automate open source security tracking with the WhiteSource CircleCI orb
- Adding Container Security Scanning to Your CircleCI Pipeline with Anchore
- Pull in your stored sensitive data with the CryptoMove orb
- Build with CircleCI, configure securely with Alcide
Private orbs on CircleCI
While thousands of developers to date have enjoyed their experience with our open source orbs, many of our customers at larger organizations have needed a way to make their orb instances private. Private orbs are available on all CircleCI plans and provide developers with increased privacy, efficiency, and collaboration across teams. This is especially useful for teams working in healthcare, finance, and other industries with high governance and compliance standards.
To learn more about how private orbs increase privacy, efficiency, and internal sharing across an organization, read our tutorial on how to build private CircleCI orbs on any organization or read our orbs documentation.
Learn how to write secure code with 7 concepts of CAS
Continuous Application Security (CAS) is a methodology that helps developers reliably build and operate secure applications and APIs. CAS transforms paper-based security policy and guidance into “security as code”.
Through instrumentation-based security enforcement, CAS enables development, security, and operations teams to work together at the pace of modern software development with scale. Learn how to use concepts of CAS here.
Adopting the DevSecOps methodology
Your team is your most valuable set of security researchers. They know your applications best and constantly work on them. Here’s how to support them:
- Implement a clear process for your teammates to report security issues
- Dedicate time to fix the security flaws that are identified
- Celebrate those who report issues openly.
Think about how long it takes for someone on your team to report suspicious activity. Do they need to post in multiple Slack channels to get attention, or is there a streamlined process to get an issue like this looked into? How do you treat those who report false positives — do you blame them for wasting the team’s time or do you hold a blameless post-mortem so that everyone on your team learns from the mistakes? All these questions can make or break your security-minded development culture.
Check out Three rules for turning DevOps into DevSecOps.
In this article we covered a number of ways toimplement standard security measures for CI/CD pipeline security. How can you use CI/CD pipelines to discover and patch security vulnerabilities? Learn more in Vulnerability Management and DevSecOps with CI/CD.
