Welcome to the DevSecOps and CI/CD security guide. Browse through each section to discover various relevant resources to ensure security of your applications and infrastructure.
What is DevSecOps?
DevSecOps is the philosophy of developing applications and infrastructure securely from ideation to deployment. It requires consideration of security risks at all stages of the development lifecycle. While DevOps teams have historically focused on automating the building, testing, and deployment of their applications, DevSecOps includes automating security practices to allow teams to increase security without losing velocity.
Secure CI/CD pipelines with DevSecOps
Secure CI/CD pipelines are at the core of daily operations for many businesses today. These processes, when set up correctly, help to keep the delivery process consistent by automating many manual tasks and providing visibility into how the software is being worked on.
CI/CD is also the place in your technology stack where your infrastructure has access to many different resources, from development and production environment to analytics keys and code signing credentials. The more resources your pipelines have access to (secure secrets, proprietary code, databases, etc.), the more important it is to keep your CI/CD system secure.
In this article, we talk about the different parts of security for CI/CD pipelines and highlight a number of ways to improve on these aspects.
Three categories of security best practices for CI/CD
We usually divide security practices into three parts that you can address with different kinds of solutions. These are:
- Secure pipeline configuration
- Code and Git history analysis
- Security policy enforcement
All three aspects are equally important. Let’s take a deeper look at these categories and ways to address them.
Secure pipeline configuration
It is possible to use your CI/CD pipeline configuration to make security issues less likely to happen.
First, safely store secrets that you use in your pipelines for connecting to databases and third-party services. On CircleCI, you have the option to use encrypted-at-rest environment variables, or to use the contexts feature. Contexts are used to provide access to environment variables across projects. Their use can also be restricted to specific security group members as defined by the organization’s administrator. Another option is to use a third-party solution to dynamically fetch secrets from their secure storage for your jobs. To ensure that none of your credentials become exposed to those who shouldn’t see them, you should never check these into your repository in plain text, even if the repository is private.
Second, for sensitive files like code signing keys we recommend adding an additional layer of isolation between your encrypted files and your repository for added security. See an example here. Keep them encrypted in your repository and store the decryption key in the environment variables or contexts, only decrypting them inside your CI/CD jobs when they are needed. This makes it even more difficult to expose or leak these highly sensitive files.
Third, you never want to leave your CI/CD environment running without monitoring it. Therefore, make sure that the containers and VMs used for your pipelines are destroyed after jobs that require sensitive information finish running. On CircleCI, this happens automatically for all supported platforms including Linux on Docker and Machine, macOS, and most recently, Windows.
Fourth, pay extra attention to how your CI/CD system handles builds for pull requests that come from forks of your repository. If your build stages require secrets to work, forked pull requests might gain access to them through your pipelines. On CircleCI, the default behavior is to never pass on the secrets to the forked pull requests to prevent leaking your credentials.
By addressing these four aspects you can significantly reduce the risk of security incidents due to your CI/CD pipeline configuration. Next up, code and Git history analysis.
Code and Git history analysis
We all love Git for its comprehensive history of changes, being able to look through the history of the project with a simple CLI command and see who has most recently changed a line of code, among other benefits. This also means that any sensitive information that makes it into the Git history can be accessible to attackers if a Git repository is exposed, even if the most recent state of the repository doesn’t contain the secrets anymore.
Trufflehog and GitLeaks are two tools that can help you identify secrets that have been committed to the codebase so that you can deactivate and replace them. These two tools will also scan your Git history for traces of secrets that your team might have added to the repository in the past.
Once your Git history is clear of secrets, you can proceed to the next level: making sure that your current revision doesn’t contain any vulnerable dependencies.
Static Application Security Testing (SAST) techniques can look through the application in your commit and analyze its dependencies. If any dependencies contain any issues or known security vulnerabilities, your commit will be marked as insecure and won’t be allowed to proceed to deployment.
Dynamic Application Security Testing (DAST) techniques go one step further and spin up a copy of your production environment inside your CI job in order to scan the resulting containers and executables. The dynamic aspect helps the system catch dependencies that are being loaded at launch time, for example, as those won’t be caught by SAST.
Security policy enforcement
Some security aspects can’t be statically checked based on known vulnerabilities, but rather are specific to your particular company, and therefore need to be codified as policies. These can take the form of automated or manual compliance checks.
Certain tasks like reviewing the list of all accounts that have access to your repositories, or making sure your onboarding and offboarding processes are in sync, will be manual. We recommend that you set reminders to perform these regularly.
Some tasks actually can be automated quite easily. For example, third-party services can provide a convenient way to codify a set of rules that will be matched against in your CI pipeline, e.g., proving compliance with the regulations that govern your data. If there is a mismatch, the build will fail.
Securing your CI/CD pipelines with CircleCI orbs
CircleCI orbs are shareable packages of CircleCI configuration to use in your builds. Orbs define reusable commands, executors, and jobs so that commonly used pieces of configuration can be condensed into a single line of code. Orbs are reusable, shareable, open source packages of CircleCI config that enable the immediate integration of these services. With orbs, you get an out-of-the-box solution for securing your pipeline.
Looking for orb-specific resources? Check out the following blogs:
- DevSecOps and CircleCI orbs: secure your CI/CD pipeline
- CircleCI for AWS: Deep integration, security, and flexibility
- Automate open source security tracking with the WhiteSource CircleCI orb
- Adding Container Security Scanning to Your CircleCI Pipeline with Anchore
- Pull in your stored sensitive data with the CryptoMove orb
- Build with CircleCI, configure securely with Alcide
Private orbs on CircleCI
While thousands of developers to date have enjoyed their experience with our open source orbs, many of our customers at larger organizations have needed a way to make their orb instances private. Private orbs on CircleCI Scale plan provide developers with increased privacy, efficiency, and collaboration across teams. This is especially useful for teams working in healthcare, finance, and other industries with high governance and compliance standards.
Learn how to write secure code with 7 concepts of CAS
Continuous Application Security (CAS) is a methodology that empowers developers to reliably build and operate secure applications and APIs by transforming paper-based security policy and guidance into “security as code”.
Through instrumentation-based security enforcement, CAS enables development, security, and operations teams to work together at the pace of modern software development with scale. Learn how to use concepts of CAS here.
Adopting the DevSecOps methodology
Your team is your most valuable set of security researchers because they know your applications best and constantly work on them. Have you implemented a clear process for your teammates to report security issues if they find any, and do you dedicate time to fix the security flaws they find? Do you celebrate those who report issues openly? In many companies, the answer isn’t an immediate “yes” to these, and it should be.
How long does it take for someone on your team to report suspicious activity? Do they need to post in multiple Slack channels to get attention, or is there a streamlined process to get an issue like this looked into? How do you treat those who report false positives — do you blame them for wasting the team’s time or do you hold a blameless post-mortem so that everyone on your team learns from the mistakes? All these questions can make or break your security-minded development culture. [Check out Three rules for turning DevOps into DevSecOps for more information]
In this article we covered a number of ways that can help you implement standard security measures for CI/CD pipeline security. How can you use CI/CD pipelines to discover and patch security vulnerabilities? Learn more in Vulnerability Management and DevSecOps with CI/CD.