[INT. - HOTEL BALLROOM] A cold, windowless conference room at the Luxor Hotel in Las Vegas during the summer of 2019. Several dozen people, most wearing black hoodies, are hunched over laptops whispering about default system passwords and hunting for deleted (supposedly) password.txt files in Docker containers. Hedphelym by Aphex Twin plays softly, menacingly.
Nope, this isn’t the legendary DefCon Capture the Flag contest. But it’s designed to emulate that event. And it’s how we decided to teach secure coding to our entire engineering team at CircleCI.
“Who knew security training wasn’t just 90’s clip art with the bad guys wearing ski masks while typing?” -CircleCI VP of Platform Michael Stahnke
How to teach security
Secure code training is one of the first things Chief Technology Officer Rob Zuber asked me to handle when I started as CircleCI’s first security engineer a couple years ago. He wanted it to be fun. He wanted it to create tension in people’s chests. A few years earlier, he’d taken part in a security training event at Google. During an exercise at the event, he discovered a vulnerability that was wide open on his service. His heart started pounding as he raced to patch it. That pulse racing experience changed the way he thought about software. He wanted all of our developers to experience it.
We looked at every training imaginable, from online courses to flying the entire department to a security conference as an offsite. None of them promised that same visceral experience.
As an organizer and MC of the Bay Area OWASP MeetUps, I get pitched lots of presentation ideas. One came from a small company based out of Hungary called Avatao, which offers secure code training. I gave some of their training modules a whirl and immediately knew that, not only could they present, we’d found a tool for CircleCI training. This was hands-on breaking things, with some guidance. This felt different than manual static code analysis games and similar things I’d been aware of. It captured concepts up and down the stack, from OWASP Top 10 application matters to backend technology like Docker, and focused on breaking into those things.
The training platform is designed by three security researchers who, not coincidentally, have been finalists multiple times at the DefCon CTF. It includes a couple hundred modules on everything from binary code exploitation to SQL injection to language-specific matters. And it’s extensible. If we wanted something they didn’t have, they’d build a module or help us write one.
The perfect opportunity
While my security training research was starting to bear fruit, our Engineering and Product teams were busy planning a week-long offsite in Las Vegas. I was given four hours of prime time after lunch on the second day to run a game, and knew this would be a great opportunity to invite Avatao to come work face-to-face with our team. Together with a few others from our security team, we selected a list of 12 modules focused on the topics most relevant to our engineers’ daily lives, things like Docker secrets, OWASP Top 10 exploits like Cross Site Request Forgeries (which we’d dealt with internally), a Vault tutorial and default passwords. For good measure, we threw in replications of a couple of real world hacks like the Facebook Imagekick.
Shopping for prizes in anticipation of the event was a blast: a lock picking book, a set of lock picks, clear lucite handcuffs and padlocks that revealed their locking mechanism.
After lunch on that second day of the offsite in Las Vegas, the Avatao folks installed two large scoreboards at the head of the room and kicked off the competition with all participants breaking into pairs. Pair programming is a big part of the culture at CircleCI so we utilized that for the event, too. As a twist, we intentionally paired engineers with people and teams they didn’t typically work with and organized them based on experience level. The pairs were assembled so that each had an average skill level of E3/4, to prevent one senior group from stomping everyone.
Within a few minutes of event kickoff, someone pointed out it was too quiet. Did I have any music?
As it so happened, I did. Back when I was a private investigator, I put together a playlist of evil music (think all of Michael Mann’s movies, American Gangster, Black Hawk Down, etc.) for a big project of hunting assets associated with the Bernie Madoff Ponzi scheme. I plugged my phone into the sound system and, gotta say, a room full of clicking keyboards made that music sound cooler than it ever did on my headphones.
Half an hour in and I knew it was a success. Every engineer was focused on their screen, not a single person was sitting back and chatting, and there was a sense of competition in the room. Blessedly, the two scoreboards on projection screens showed every team was making progress. Our security team was on call as teaching assistants, but they were huddled together in a corner talking amongst themselves. No one was having a problem.
At the end of two hours, we handed out prizes, conducted a Q&A with the Avatao folks and then broke into groups of six engineers so they could discuss both what each person learned as well as what was the most applicable to our internal processes. Finally, everyone moved to an adjoining room for cocktails, coffee, and lock picking exercises.
Lessons learned, securely
So what did people report out of their six-person breakouts?
The event took away some of the mystery around how specific issues like cache poisoning happen. Reverse engineering real-world hacks like Imagetrick is fascinating. The modules were challenging, but not discouraging. Security reviews are really important. And, virtually every group asked for specific modules to be added on Kubernetes and API validation.
“Capture the flag was cool,” said Software Engineer Breon Knight, who paired with a Principal Software Engineer. “It was interesting to see it from a principal level engineer’s mindset.”
Software Engineer Jacqueline Garcia said it was interesting to see security bugs firsthand and to spend two hours discussing them with a teammate. That made them think differently about implementing security into coding practices.
“What I enjoyed the most was the collaboration involved,” she said.
- Focus on modules that are the technology your audience uses every day. Just because the security team is interested in an esoteric exploit doesn’t mean everyone else will be.
- Keep the exercises short for quick wins so people don’t get bogged down. It’s important that people feel like they can do security rather than driving home the idea that security is hard and should be left to specialists.
- Put together a group of engineers from across the department who will help shape the curriculum by trying it out in advance. This not only aligns the modules with needs, it creates teaching assistants during the event who can triage problems.
- Security is more than just code. Add things on both sides of the programming like lock picking to keep the fun quotient high.
If you’re looking for ways to secure your CI/CD pipeline, check on our post on security best practices for continuous integration.