Is my CI pipeline vulnerable?

Your continuous integration (CI) pipelines are at the core of the change management process for your applications. When set up correctly, the CI pipeline can automate many manual tasks to ensure that your application and the environments it runs in are consistent and repeatable. This pipeline can be an integral part of your security strategy if you use it to scan applications, containers, and infrastructure configuration for vulnerabilities.

However, the pipelines themselves can contain vulnerabilities that could potentially compromise all your systems. Fortunately, with a bit of care and shift in the design “culture,” you can find and mitigate CI pipeline security problems.

In this article, we are going to discuss a technology-agnostic approach to discovering and mitigating these pipeline vulnerabilities, as well as ways to ensure security features from design to deployment and beyond.

Discovering vulnerabilities

When you look at a complete CI pipeline, you usually see several components that work together to provide an efficient integration and deployment process for your applications. These components need to trust each other to deliver the right information through the process steps to ensure the desired result is fast and consistent every time. Because pipeline components have different dependencies and requirements, there may be issues that attackers can exploit.

Luckily, most vulnerabilities relate to the configuration or management of your pipeline. Some of the most common security issues or risks that are likely to exist in your pipeline include:

• Shared, open, or compromised source code repositories
• Build server and container misconfiguration
• Lack of updates to operating systems, runtimes, and tools
• Exposed secrets, keys, and credentials in code repositories
• Lack of monitoring tools or regular maintenance
• Poor management of access controls and security settings of components

Addressing vulnerabilities

Most of the issues on the list of vulnerabilities also exist within the applications you build, so finding and addressing these vulnerabilities should be somewhat familiar. Just like with an application, it is essential you study and evaluate your pipeline’s attack surface to come up with a set of best practices to protect it.

Using the list above as a starting point, the following steps will help minimize the risks associated with these security concerns:

• Lock down your source code repositories. Minimize the number of people who have access to approve pull requests that are directly linked to your pipeline, and use a rigorous code review cycle.
• Adopt infrastructure as code (IaC) practices for your build servers and, where possible, try adopting short-lived containers with a focus on built-in security.
• Regularly audit and update the operating systems, runtimes, and build tools within your pipeline. A regular schedule of security patches and bug fixes helps reduce exposure to bugs and flaws in the libraries used to build your applications.
• Remove secrets, credentials, and keys from code and adopt a secret manager such as SecretHub, Fortanix, or AWS Systems Manager Parameter Store.
• Adopt auditing tools that allow you to track changes made to your pipelines. Use code analysis tools that scan code for common exploits to ensure these are corrected at the source.
• Implement access controls and security policies across the entire pipeline to block unauthorized or anonymous access.

What is DevSecOps?

The remedies I have described rely on the adoption of a DevSecOps mentality. DevSecOps is a set of additional practices that aims to move security concerns in more common DevOps processes closer to development, by “shifting left.”

The general idea is that a large portion of security concerns and risks can be addressed during design and development. In these earlier phases, developers can plan for, and automate, security from the start, rather than relying on testing security after deployment. It is this shift in culture, where everyone involved in software delivery thinks about the security of your application, that allows secure application development at speed.

What can you do to start expanding your DevOps practice into a DevSecOps practice?

You do not necessarily need to add expensive specialist tools to start moving towards this practice. Some common strategies are:

• Encourage all stakeholders to start developing an understanding of modern security threats in application development.
• Develop processes for the different stakeholders across the development cycle to respond to the various security incidents.
• Leverage your existing DevOps communication channels to allow people to share security issues and risks quickly, to speed up resolutions. Implement regular security and compliance audits across the development lifecycle as part of the normal software delivery process.
• Adopt appropriate security tools that can help you enforce security and compliance polices, and that can check for security vulnerabilities across your code and pipelines.

Code vulnerabilities

A core component of the DevSecOps philosophy includes designing and testing secure code as it is written, or at the source, on a regular basis. This process is called static application security testing (SAST). There are a number of tools that can help automate SAST.

SAST is a form of testing for the internal structures of the application, generally against the type of vulnerabilities like those found in the OWASP Top 10. Tools like Snyk, WhiteSource, and NowSecure can run against your code repositories as part of a pull request or during development, catching security issues before they even make it into your pipeline.

The more traditional dynamic application security testing (DAST) is a form of testing that runs against an application during execution, without details of its internal structures, for exploits that an attack can use.

Ideally you would use both static and dynamic application security tools in your development lifecycle. While static testing will not find exploits related to environment-based security issues like dynamic testing would, it will catch a lot of security issues early, before they become more expensive to fix. Combining both of these forms of testing, and automating them into your pipelines, will provide a comprehensive layering of vulnerability scanning.

Adding SAST tools becomes a key part of the long term “shifting security left” mindset, raising security issues and risks with developers as they code or before they merge. This helps developers become more aware of common security issues and allows them to form good habits on writing code that passes security checks on the first try.

Adopting a SAST tool and incorporating the results into code reviews allows developers to get up to speed quickly while still using DAST tools in QA and production to achieve more rigorous security testing.

Next steps

Consumers of your application deserve and expect it to be performant, usable, and secure, especially if you store their data, or they rely on it for day-to-day operations. By securing the pipelines that build your application and minimizing the risk of pipelines being compromised, you can improve the speed and reliability of your deployments.

Layering a “shift left” attitude to security as a practice and adopting security tools in your development processes remove much of the security risks associated with your application. This combined approach allows you to take advantage of the DevOps principles of automation and repeatability.

Start by making sure that your CI pipelines are secured and maintained. If you are not doing DevSecOps already, why not give it a try? CircleCI offers numerous security integrations to combat vulnerabilities we have discussed, and it does not cost a thing to get started.