How to secure your CI pipeline

Many enterprises still struggle to get security right. To protect their business, it is critical they focus on security during the entire infrastructure and application lifecycle, including continuous integration and delivery (CI/CD). CI/CD is a great place to implement different security layers because CI/CD is all about automation. Automating your development processes reduces the number of holes in the layers by removing the insecure human element.

As developers become more autonomous and transition to a DevOps way of working, more people will need access to production systems. These developers are not security experts, and security teams are not developers. With increased autonomy, development teams must learn how to deliver quality applications securely.

Secure CI/CD pipelines for the enterprise

These days, instead of having just a firewall or outer security edge, most organizations build in multiple defense layers to minimize security risk. One of the best places for developers to add security mechanisms is in CI/CD pipelines. These pipelines can include many fully-automated protective measures to cover many of your security requirements.

In this post, we will describe security mechanisms you can employ to better secure CI/CD pipelines while enabling developers to push their code, quickly and safely.

What is the Swiss cheese model?

The range of security measures many organizations use typically provide layers of protection. However, each of these layers contains weaknesses (holes) in its defensive structure. We call this the “Swiss cheese model.” It is a convenient way to think about the layers and the holes in each layer. Ideally, the layers line up so that the holes in one layer are not in the same place on the next layer. In other words, the vulnerabilities don’t go all the way through. You want your sandwich continuously covered with security cheese.

The Swiss cheese model was introduced by James T. Reason and Dante Orlandella in research on the subject of “cumulative act effects” and human error in several engineering disasters, including the Challenger space shuttle.

Applying the Swiss cheese model to software applications and infrastructure can help you understand why additional layers of defense still fail. Looking at each layer of defense as a slice of Swiss cheese shows how security still allows unintended system access if those holes line up exactly. In the Swiss cheese model, any system with human involvement can still fail. Quite simply, humans make mistakes.

The human element means that these layers of defense are not just technical in nature. Social engineering attacks target employees directly to get them to reveal passwords or personal information. Awareness campaigns and training help these employees to recognize such attempts. Still, humans are the weakest link in the chain.

Instead of trying to prevent humans from making mistakes, it makes much more sense to take humans out of the equation entirely. By automating security, enterprises reduce the risk of holes lining up in their security layers.

Use CI/CD to strengthen security layers

Implementing CI/CD enables you to remove manual, human access for certain tasks. For example, after automating virtual server creation, your engineers no longer need permissions to create these servers manually. Extend this approach to all your infrastructure and applications and you can ensure your entire environment is built consistently. Be sure to implement proper security mechanisms for those pipelines, such as granting only least-privilege permissions.

Other options for securing CI/CD pipeline layers include:

• Routine scanning
• Security testing
• Monitoring

Scanning methods for securing pipelines

One benefit of fully deploying infrastructure and apps with CI/CD is that your entire environment is described in code. You can automatically scan this code, as well as other artifacts that are part of your CI/CD pipelines, to address security concerns without much effort.

The different phases typically associated with CI are building, testing, and publishing a deployable artifact. In that test phase, you can implement security scans in addition to your unit and system tests.

Many free and open-source tools are available for different types of scans. Scans you should implement into your pipelines are:

• Source code scans
• Container scans
• Infrastructure scans

Scanning source code

Source code can contain different types of vulnerabilities. If external data such as user input isn’t properly sanitized, hostile data can lead to vulnerabilities such as SQL injection or cross-site scripting. Advanced scanning tools can detect such patterns in your source.

External dependencies such as libraries included in your application through, for example, maven or npm can also contain such vulnerabilities. Use dependency vulnerability scanning tools to analyze dependencies for known CVE vulnerabilities.

Finally, scan your source code for the usage of plain-text secrets such as database passwords or tokens. Such secrets must always be encrypted or stored in an external secret manager solution to minimize leaking these values.

Scanning containers

Many organizations deploy their applications through containers. Containers follow a layered pattern where the application is built on a public container such as from the Docker Hub. Scan each layer to lessen the chance of deploying known vulnerabilities to your infrastructure.

Scanning infrastructure

Infrastructure can be configured in many different ways - some configurations are more secure than others. Whether or not to enable encryption is often as simple as a checkbox in the Cloud. Endpoints can be configured with or without encrypted traffic (HTTPS). Ensure your infrastructure as code complies with defined security standards and best practices in your organization through infrastructure security scans.

Testing to secure CI/CD pipelines

CI/CD pipelines can execute several different security tests as part of their workflows. Such tests typically fall into two categories.

• Static application security testing
• Dynamic application security testing

The first category, static application security testing (SAST), includes most of the scanning tools described in the previous section. Such scans find vulnerabilities early in your pipeline before deploying anything to an actual runtime environment. This layer of defense prevents security issues from reaching your production environments.

Apart from preventing issues in your environments, this also provides early feedback for your engineers. They can quickly mitigate any concerns while actively working on the code. This practice is called shift left: detect and mitigate issues as early as possible in the software development lifecycle.

The second category, [dynamic application security testing (DAST)], scans the now-deployed infrastructure or application. This second scanning layer identifies some similar vulnerabilities as the SAST scans as well as different vulnerabilities. You can run these scans on a staging environment - again, preventing issues from ever reaching your production environment - but you should also run these scans on your production environment.

Implementing a combination of all these tests helps you achieve multiple layers of defense. Best of all, all these scans can be fully automated. You can learn more about SAST and DAST scans in SAST vs DAST: What they are and when to use them.

Monitoring and auditing to keep pipelines secure

Do not overlook the CI/CD pipeline itself when it comes to security. The pipeline is a critical system with permissions to make changes to your infrastructure and applications. Vulnerabilities in this system can compromise your entire environment. Monitor and audit your pipelines to ensure they work as expected and are secure.

Risks associated with your pipelines include access keys that haven’t been rotated and giving the wrong people access. Monitoring tools check for issues and consolidate alerts to provide the correct insights and ensure vulnerabilities are addressed. They also alert the right people in case of issues.

Consider implementing automated remediations such as revoking access and disabling access keys to improve your security. Use these auditing tools to review any changes to the pipeline itself and implement a process, including automated checks and manual reviews, to protect pipeline quality.

Wrap up

We looked at many different methods you can employ to help secure your infrastructure and applications.

• Implementing several layers of defense works well. Remember the Swiss cheese model and be aware of potential holes in your security.
• Understand that humans are the biggest risk when it comes to security and automate your processes as much as possible.
• Scan, test, monitor, and audit across your infrastructure and application lifecycle. Remember, too, not to overlook managing and securing the pipelines themselves.

Now that you understand the benefits of pipeline security and where CircleCI fits in, why not give it a try? Sign up for your CircleCI free trial today to simplify your DevOps and get your team moving to the modern cloud.

Need more information on CI/CD security? See the Ultimate guide to CI/CD security and DevSecOps.