About a year ago, our engineering organization realized it had a problem. CircleCI was in hyper-growth mode and needed to hire more engineers… and we needed to do it fast. But our current process was taking far too long. We knew the process could be improved, we just had to figure out how.

So, as we often do at CircleCI, we began experimenting. We prioritized making more time for interviews, we tried outsourcing technical interviews to a third party, and we tried out new and different types of questions for candidates. When we still weren’t finding the right talent fast enough, we iterated on our process again and again.

After conducting more than 6,000 engineering interviews in one year, this is what I’ve learned about the best way to find top engineering talent and how to scale that process for a rapidly growing company.

Make more time for interviewing

The first problem we identified is that our interview process was simply taking too long. We were losing out on top candidates because they couldn’t wait weeks to complete the interviews. The initial solution we came up with was to re-schedule some of our internal meetings to make more time for interviews. This allowed us to move candidates through faster but had the adverse effect of sucking up all of our internal team’s time and energy.

This is when we first decided to try outsourcing technical interviews to a third party. It took the strain off our internal team because candidates could schedule interviews any time of day or night, and the third-party company had the technical know-how to accurately assess the candidates. Still, our team had to watch every recorded interview to see if they agreed with the assessment. It was also difficult to modify or iterate on the interview process because we didn’t own the whole thing. And, importantly, some candidates did not like that we used a third party for the technical interview portion and dropped out of the running because of it.

Use a third party to support hiring but don’t outsource the entire process

It was clear we needed to own more of the interview process so we could have control over the questions, exercises, and assessments, but we didn’t have the bandwidth to run the entire thing ourselves.

That’s when we decided to move our technical screening to Pair Programming with a platform called CoderPad. CoderPad doesn’t do the technical interview for us, they simply provide an environment for the candidate to pair with us on code. We still format the questions and conduct the interviews ourselves, which gives us the ability to evolve the process as we learn what works and as we scale.

Design questions that will give you the signal you want for that role

For some roles this takes 5 minutes; for other roles, I spend hours on this. The more work you do to create thoughtful questions that send a good signal, the better the results are. When I’m interviewing, I’m not personally vetting coding skills. I’m usually vetting higher-order skills like the ability to work with a team, navigate an organization, or motivate others.

I often start by asking things like:

  • What do you want out of a new role that you’re not getting today?
  • How do you measure success in the role of [whatever they applied for]?

Sometimes I get into more specifics but usually, it’s better to keep the conversation pretty high level. I also use the same questions for all candidates applying for the same role to help mitigate bias and ensure a level playing field.

Learn how to avoid bias in interviews and encourage candidates to advocate for themselves

Don’t ask abstract questions. Ask about the skills needed to do the job.

For some roles, we describe a scenario in which a system has an operational failure and have the candidate walk us through troubleshooting and debugging it. For others, we might have them enhance a system based on new amounts of traffic or load. We ask about the things you need to be able to do in the job because it’s a far better indicator of that person’s problem-solving skills.

I often ask something like “What are you most proud of in your career or that you’ve built?” That question usually comes early in the interview to establish rapport and get the candidate to talk positively about something. Sometimes I’ll ask more open-ended questions like “What’s difficult about building, operating, or validating distributed systems?” Those questions often give me a lot of signal on experience level.

Culture isn’t about wanting to hang out with somebody; it’s about work styles

Can this person work in the open? Can they pull others with them via influence? Can they carry a message even if it’s not their favorite thing? Can they optimize globally versus locally? These are things I ask to find out if a candidate is a good culture fit. Culture fit doesn’t mean you want to socialize with that person, it means they have work styles that are compatible with your team.

Most questions I ask start with “tell me about a time” or “give an example of a time when you…” I want specifics. If I’m interviewing you, I want to understand if you made something good happen or stood next to people who did. At the same time, do you take all the credit or do you talk about the success of a team? When you have failures is it always somebody else’s fault or do you own your mistakes or problems?

Look for people who are curious

At CircleCI, we want curious engineers. If you stop looking for answers after one layer, you might not really understand what is happening or why. For example, during an outage, if you find that the system went down because the database ran out of connections, you haven’t fixed anything. You can create more connections or restart the database but do you know why it is out of connections? Is it due to long-running queries? Is a connection pool algorithm broken? Are we not closing connections when we’re done with them? Did we increase a timeout on something?

There could be dozens of reasons for an outage. Let’s find out which one it was. A lot of people talk about finding the “root cause” but I see that as a problematic label because it often encourages people to stop looking. There is always another layer deeper to look into. Those are the kind of engineers I’m looking for.

Interviews are not combat

Remember, you’re not there to try to show that you know more than the candidate or to see where they fall over. You’re there to see if they can do the work and if you can work with them. Helping a candidate along isn’t cheating… you help out your colleagues in a work setting all the time. So, come back to whether or not the candidate has the foundational knowledge the role requires. And again, can you work with this person?

Interviews are stressful so no matter what, you should ensure every candidate has a great experience. If they don’t get the job, you want them thinking “darn, I wanted to work there. That place seems awesome.” Maybe they’ll apply again under better circumstances, maybe they’ll refer somebody else. In the end, they put themselves out there so let’s recognize that and deliver the whole process with respect.