[INT. - HOTEL BALLROOM] A cold, windowless conference room at the Luxor Hotel in Las Vegas. Several dozen people, most wearing black hoodies, are hunched over laptops whispering about default system passwords and hunting for deleted (supposedly) password.txt files in Docker containers. Hedphelym by Aphex Twin plays softly, menacingly.

That was the first paragraph of last year’s blog post about the CircleCI Capture The Flag (CTF) competition in Las Vegas. This year’s competition was a little bit different.

The journalist Hunter S. Thompson once reused a Superbowl story from the previous year — updating the teams, names and location — because the events were so similar. This year’s CircleCI CTF wrap-up should have been that easy: new winners, different location, etc. Needless to say, it was far from it.

A 100 person security training… online?

A global pandemic isn’t a normal problem. But despite the challenges, we managed to put together a successful CTF competition — entirely online. And while security training is incredibly important for engineers at every company, the team-building events this year were more important than ever. Not only did they create connection during an isolating time, they also gave our security practices a confidence boost.

As the person overseeing our Security Team and SOC 2 certification, I don’t have an option to punt on security training for engineers. It’s written into our report and gives me some peace of mind that our engineers are leveling up regularly in such a critical discipline. Plus, as a member of the CircleCI COVID Task Force, I was constantly hearing from employees about the struggle to stay connected with colleagues.

We had to move forward with an entirely online CTF event.

Recreating ad-hoc team-building remotely

Last year’s event in Las Vegas was so. much. fun. A couple of brand new engineers won first place. People wrote code that had no purpose beyond fun. After handing out gag magic prizes, lock pick sets, and copies of Violent Python to the winners, everyone decamped to a lounge area for drinks and coffee while one of our engineering managers taught a lock picking class. People stood around, talking about whether it was too obvious to guess “admin admin” or whether that was a sign they had a suspicious personality, and how to make the event better the following year.

So, how do you recreate that during a pandemic?

Why tight relationships are a win-win for security

For starters, we stuck with the Avatao platform because the training modules are excellent, most engineers were familiar with the system, and they now have more than a hundred available, which made it possible for us to create Front End, Back End, and Infrastructure tracks instead of a single track like last year.

Front End

  • Do not share your secrets - Part 1 (JS Deobfuscation) (10 minutes)
  • Do not share your secrets - Part 2 (JS Deobfuscation) (20 minutes)
  • Sadness 1 (Login SQLi) (5-10 minutes)
  • NoSQL Company (NoSQL Injection: MongoDB + PHP) (30 minutes)
  • Secure Bank (IDOR) (5-10 minutes)
  • Where The Hack is He? (60 minutes) (It’s a little bit long, but it’s a fun programming challenge - especially if they’re working in teams)
  • Alert Me 1 (20 minutes)
  • Noncebook (5-10 minutes)

Back End

  • Sadness 1 (Login SQLi) (5-10 minutes)
  • Login Screen (Login SQLi with filters) (15 minutes)
  • Docker Build Secrets (Docker) (30 minutes)
  • NoSQL Company (NoSQL Injection: MongoDB + PHP) (30 minutes)
  • Secure Bank (IDOR) (5-10 minutes)
  • Secflix groups 1 (Authorization) (40 minutes)
  • Login With Identity Provider (JWT: OpenID) (30 minutes)
  • Where The Hack is He? (60 minutes)

Infrastructure

  • Docker Build Secrets (Docker) (30 minutes)
  • No Secrets For You (Security Misconfiguration) (20 minutes)
  • Dog Pictures (3rd party) (40 minutes)
  • Login With Identity Provider (JWT: OpenID) (30 minutes)
  • Simple Recipes (Security Misconfiguration) (15 minutes)
  • The Broken Algorithm (Crypto) (35 minutes)
  • Fake Andromeda Certificate (Crypto) (50 minutes)
  • You vs zh4ck (Fun Golang “reverse engineering”) (40 minutes)

We also stuck with our team structure to emphasize pair programming, cross-team interaction, and level setting. Engineers were paired based on experience level, to prevent one senior group from stomping everyone, and to pair folks who don’t normally work together. Tighter relationships are a win-win for security. They build trust and break down communication barriers so people feel more comfortable raising issues.

Dialing up the fun for our distributed team

Our biggest challenge was scheduling around these parameters:

  • More than 100 engineers and managers.
  • Participants live in EMEA, US and APAC time zones.
  • All the games must happen on Monday, Tuesday or Wednesday to work for everyone.
  • Every other engineering meeting had to be rescheduled.

While security training is important, team-building felt more important than ever given the forced isolation everyone’s been working in. Dial up the fun. That’s where The Go Game came in.

About 1.5 hours were spent on security training, with the rest spent on team-building events. There were a few ice breaker trivia questions (where does the Release Engineering Manager like to vacation?). But we customized everything around security with questions like “Which hacker movie does the CircleCI security team dislike the most? (Answer: Swordfish).” Last year, the room was silent and that was a good thing. No one was walking around, leaving or talking rather than learning. But as we all know, silence on video calls is the worst. So having a host keep the action moving was key.

“It was just a really fun thing to do,” said Lena Reinhard, Vice President of Product Engineering. “Working with someone I never work with, solving puzzles together, and learning weird fun facts about other people was quirky and nice.”

Not everything worked as planned, though.

The first game suffered from an authentication problem. The entire experience involved using four different platforms (The Go Game, Avatao, Slack, and Zoom) at the same time. The infrastructure track was unintentionally hard, like walking into a shelf with your forehead.

Despite that, people appreciated the effort to hack a solution like this together and embraced its purpose. For prizes, we handed out gift certificates to Browser’s Den of Magic, which is Canada’s oldest magic shop. COVID shut it down and supporting it seemed like the right thing to do.

After the event, our feedback survey garnered a 50 percent response rate and, even better, 17 engineers asked to be part of a working group that will set up next year’s event. Common themes for next year’s event include better hints for the host so she can help people who are stuck, video solutions to all the problems that participants can watch afterward, and no.more.php.

Until Next Year

I watched 18 hours of these games in real time and can honestly say these were some of the best video calls of the pandemic. People were laughing at each other, leaning back in their chairs with delight and generally acting looser than normal.

That said, I can’t wait until we can do this in person next year. Until then, it’s great knowing our engineers can come together under challenging circumstances, learn something important and have fun together, all while improving our security practices.