For the past five years, CircleCI has hired third-party penetration testing firms that hammer our product releases or infrastructure at minimum every 90 days. To date, we’ve done more than 25 of these. Sometimes they cover new product features and other times they cover infrastructure.
Penetration testing for CircleCI Server 3.0
We recently underwent third-party penetration testing for our server 3.0 major release. This release made Kubernetes a first-class citizen of our enterprise offering. With the major shift to Kubernetes as the underlying infrastructure that all ensuing releases would be built on, we sought out the best party we could to test the resiliency of our new foundation. Enter: Trail of Bits.
Partnering with Trail of Bits
Trail of Bits is one of the two third-party penetration testing firms the Cloud Native Computing Foundation, which oversees the development of Kubernetes, hired to test Kubernetes itself over a four-month period in 2019. Linking our work as closely as possible seemed like the best way forward.
CircleCI gave Trail of Bits a beta release of our Server product and asked them to hammer on it at will with nothing more than the instructions we’d provide to a new or existing customer. Three weeks later, they had found some astonishing holes that delayed the product release. And for that, we are extremely grateful. We were more than happy to delay the release (and are infinitely grateful to customers who were waiting on it) because the last thing we want to do is ship insecure software to anyone.
The Trail of Bits test of the major Server release is not unique. Every 90 days, one of our development teams goes through a similar drill. Other tests have included our Mac fleet, the new user authentication workflow, and our recent technology acquisition.
CircleCI’s penetration testing policy
Because we place the security of our customers as a top priority, we have a standing penetration testing policy at CircleCI to immediately fix 100% of all Critical and High findings, and then have those repairs validated before the test is considered closed. All Mediums, Lows and Informational findings are reviewed by a security engineer who works with the development team leads to assess the risks, and add to the team’s backlog as appropriate. A Medium very well might be a Medium and need to be fixed immediately too, but with more context we might feel confident marking it down to a Low and adding it to the team’s general backlog. Penetration tests of all our product offerings — cloud, Runner and server — follow the same process as do any findings on infrastructure tests.
Customers frequently ask for copies of our penetration test results. We are unable to provide raw findings for security reasons, but we do make available a customer-facing summary that provides as much detail as possible without exposing any potential secrets. To request a penetration test summary, reach out to your Customer Success Manager.
