In this series, we pulled aside folks from across our engineering department to talk about confidence. From the technical executives to folks on the ground in engineering, management and site reliability, we wanted to know what “confidence” meant to them, and how it had changed over the course of their careers. You can read the other five posts in the series from Rob Zuber, Michael Stahnke, Stig Brautaset, Jacque Garcia, and Mike Marquez.

In this interview, we spoke to CircleCI Staff Software Engineer, Glen Mailer. We hope you enjoy it.


What is your role and how long have you worked in engineering?

My current role is a Staff Software Engineer. I have been an engineer for… well, I tend to use the word developer, for a decade maybe? Yeah. Professionally for a decade. And before then I was a hobbyist.

What does it mean to have confidence as a developer?

So when you first said that the first thing that sprung to mind was thinking about software testing and releasing and deployments. That was my initial thought. And then thinking about it a bit more, I started to think about the confidence to ask questions and to get involved in all aspects of software delivery, not just churning out code.

How has tooling, or automation, changed your relationship to your confidence in code?

I think it all comes down to feedback cycles really. Throughout my career, feedback cycles have been getting better and better. And at some point, I realized “this is very important,” and I started making it a first-class goal.

And I think my interest in feedback cycles came about when I used to be a Ruby developer. That was the first ecosystem where I got into this idea of really, really fast tests. Running in a loop automatically where you make a change, you hit save and you get that TDD red, green, refactor cycle going. You really just get in the zone of making sure you see feedback, make the change, and see more feedback as soon as possible.

Did you have an interest in those fast feedback cycles before coming to CircleCI?

It’s something I had been doing a lot at various companies before joining CircleCI, when I worked as an independent contractor. I went around to companies and tried to help them get better at doing continuous deployment and iterative delivery, thinking about the customer first - not disappearing for a year to produce a product and “tada!” Leaving customers asking “what is this?”

you get used to certain things being normal and then you forget what’s normal for other people… I think it’s really important that whenever we’re giving advice about software development we state the context that the advice is valid in.

It’s really interesting, you get used to certain things being normal and then you forget what’s normal for other people. So, now and then someone will talk about scrum or sprints and I cannot fathom waiting two weeks to change my mind anymore. But in some places they’re releasing once a quarter, so taking three weeks sprints would be a big step forward. I think it’s really important that whenever we’re giving advice about software development we state the context that the advice is valid in. I think lots of people try and give universal advice, which I don’t think is necessarily a good idea.

How do you build confidence on your team? How can you trust the work your colleagues are doing?

I think it’s around iteration and visibility, mostly. It’s being able to make a series of small incremental changes that people can see as they go along and they feel part of it. You’ll have that shared understanding of how you reach the destination. I think that’s really important.

How do you move forward when you’re stuck or not sure you can achieve the result you’re hoping for?

One thing is definitely pair programming and looping more people in. And the other thing is trying to say, okay, can we bite off a smaller chunk? Can we slice this problem? Can we redefine the problem statement? Or even, can we take a step back and say, okay, well what actual thing are we trying to achieve? What is our customer actually wanting to do? And maybe there’s a completely different way to approach what we’re currently working on. I think that was sort of a habit that I built up when I used to do a lot of open source support work.

the more context you’ve got, the more you can understand what the wider goal is, and the more likely you are to take a step in the right direction - I think that does breed a lot of confidence.

I think knowing more context about the problem we’re trying to solve and the customer needs when you’re building something helps you make decisions. When you haven’t got that context, you will inevitably have to make loads of little decisions and guesses about how things are going to work. But the more context you’ve got, the more you can understand what the wider goal is, and the more likely you are to take a step in the right direction - I think that does breed a lot of confidence.

How has bringing in customer opinions into product or feature development built your confidence over time as a developer?

User research is something I’ve gotten really into over the last few years - which is us saying “this is what the analytics data says, this is all the feedback, and these are the problem areas that customers are having.” From that, can we figure out: what does the customer need? Not what does the customer want.

It’s also about building the confidence to keep asking those questions. It’s fortunate that I ended up in a situation where I think I was in quite a comfortable job so I could ask really dumb questions - it was a very psychologically safe environment, so I got into a good habit.

I will always ask a clarifying question. Even if I’m pretty confident that I know what the answer is, I will still often ask the question to have the speaker say it to the rest of the audience as well

I will always ask a clarifying question. Even if I’m pretty confident that I know what the answer is, I will still often ask the question to have the speaker say it to the rest of the audience as well, just to say, okay, we’re all having this shared understanding. At one company, I remember asking questions like “well, why did we do that?” “Why don’t we do that?” Just being willing to stick your head up and go, “well hang on” is an important skill.

What were some pivotal experiences that changed or improved your confidence level as a developer?

I remember quite a few years ago now, I was reading up about feature toggles and I was like “Oh that sounds like a good idea.” And at that particular company we had reasonable leeway to try that sort of thing as long as we delivered on the business metrics. So, I put some feature toggles into the system - just the ability to ship code into production and then see how it behaved in production built confidence.

At that company we always really struggled with test data. Staging was never really representative, only production was, and they were developing that muscle for how to use production safely.

And I think I realized much later that at a previous company, the main system that we used was really a CMS where we could put components in and move things on the page. Whenever we built a new feature, we built a new component and then separately we would add it onto the page. And looking back I realized that was a form of feature toggling where we would build it, put it on the wrong page, have a look at it, and then we’d go put it on the right page.

Do you ever get imposter syndrome? How do you deal with it?

Do I sound very arrogant if I say no? Imposter syndrome is not something that I really worry about. But I do try and make sure that I’m very careful with my language. I try not to make absolute statements. Like when I’m telling an anecdote about an experience, I make it clear that this is something that’s happened in the past - this is my opinion, but formed based on my experience. I make that very clear. When it’s something I know to be fact or something I’ve seen in research data, I’ll call it out.

Then again it comes back to this provided context. I try to say, “I think this because of this,” but I want to make it easy for someone to disagree with me and have a constructive discussion about my opinion.

Has your relationship with confidence changed over the course of your career? And if so, how?

I think I definitely got a lot better at iterating and writing automated tests. I’ve got this a lot from the Ruby community - thinking about BDD and TDD. It’s understanding “what problem are we trying to solve” and “can I write a test which states the problem I’m trying to solve.” I really use that to drive the code that I’m writing.

What advice would you give to your younger self?

“Ship it,” I think that that’s the advice I would give. And I don’t just mean push any old rubbish out into production. But if your goal is to ship code into production and get it into the hands of users, then you should work backwards from there to figure out, “how do I do that safely?” The first thing in the forefront of your mind should be “how do I get this out there in front of people?”

“Ship it.” The first thing in the forefront of your mind should be “how do I get this out there in front of people?”… I think “ship it” is really about the smallest step you can take. And you have to try to get used to that sort of incremental approach.

I think the main thing that helps is pushing things out in small increments - so we’ll push a feature and roll it out to a small number of customers, for example. Recently, when we were rolling out secrets masking on CircleCI, we trialed it on a couple of internal projects and we found some bugs there and fixed them. We then rolled it out to 1% of customers, nothing happened. Then 3%. Okay. Then something happened - so we collected feedback and brought the percentage of customers using it down to zero. Then, we fixed those bugs and we tried it again. So I think “ship it” is really about the smallest step you can take. And you have to try to get used to that sort of incremental approach.

Is there anything else you’d like to add?

I think of CI and CD as quite different, so solving quite different problems. But CD is about consistency. It’s about orders. It’s about repeatability, it’s about speed. It’s about not copying three files with an FTP. Whereas CI, to me, is about checking that baseline. It’s “here are the things that you might’ve forgotten that a computer can tell you about.”

I read this concept from, I think it was a ThoughtWorks deck a while ago. The test time budget. So you say, right, my budget is five minutes and I can spend a max of five minutes running my CI pipeline. On CircleCI, you can parallelize really heavily and do a lot in five minutes. The idea is “I don’t want to wait too long, I’ve got five minutes to spend, how am I going to spend it? What’s most valuable to spend my time on?” And if you’re trying to do really realistic, production-quality, testing with production-quality data, trying to catch every little edge case, you’re not going to fit that in that five minute project. So what you need to do is exercise that testing-in-production muscle.

by exercising that muscle of testing-in-production and continuously, iteratively rolling out changes, when you do have a big problem, you’ll be prepared because you have practiced on all these little problems day to day.

You’re always going to have bugs in production. So if you focus all your energy on avoiding those, when you do get the bug in production, inevitably you won’t know what to do. Whereas by exercising that muscle of testing-in-production and continuously, iteratively rolling out changes, when you do have a big problem, you’ll be prepared because you have practiced on all these little problems day to day. You’ll have your architecture assembled so that when there is a bug, it doesn’t take the whole website down.