In this series, we pulled aside folks from across our engineering department to talk about confidence. From the technical executives to team members on the ground in engineering, management and site reliability, we wanted to know what “confidence” meant to them, and how it had changed over the course of their careers. You can read the other five posts in the series from Michael Stahnke, Stig Brautaset, Glen Mailer, Jacque Garcia, and Mike Marquez.
In this interview, we spoke to CircleCI CTO Rob Zuber. We hope you enjoy it.
What does having confidence as an engineer mean to you?
Well, there’s confidence in correctness and confidence in design. Correctness is: am I confident enough to deliver this? That’s where CI/CD comes in. The confidence in my code’s correctness is actually outside of me: I’ve outsourced it. By asking the test system to validate it, I can be confident that it will work. And of course I can increase my coverage and all these other things, and that will give me more confidence.
Then there’s confidence that I’m making the right choice about how to implement something, in terms of good abstractions and design and things like that. That’s more inside of me.
And you learn that as you go. I think learning is not so much about just having more ideas, it’s about recognizing patterns and observing how things play out, so you can feel confident that something will go well enough. If I can accept that I’m confident enough in whatever I’m working on, then I can move onto the next thing.
learning is not so much about just having more ideas, it’s about recognizing patterns and observing how things play out
I think that’s actually one of the most challenging parts of building confidence. I love to talk about risk, and knowing something is only so important, and therefore, I’ve put enough effort into it to do enough work or to do the right-enough thing for the level of importance. Or knowing that this is a critical system, and I’m going to do more things to be more confident.
In decision making, we talk about this in terms of the decision making model that we’ve tried to put in place in engineering. What is the impact, the scope of impact, and what is the cost to reverse the decision? And on that two by two, you would have different points where you would include more advisors and more stakeholders to make a decision.
How has confidence changed for you throughout your career?
I think a lot about building a pattern library. Just a library of experiences that I can reflect on. I love super weird analogies, so I apologize for this:
Think about rock climbing. When people start out rock climbing on a top rope, they’re super grippy. Like, “What’s going to happen if I fall?” And then you fall, and you fall three inches. And you think, “Oh, that’s not a big deal.” And now I can actually go try things: I can jump to another hold and do these bigger things because I know that I’m going to fall a foot. And then you go to lead climbing, and that’s a whole other thing, but you take that first lead fall and realize, “Oh, it’s actually not that big a deal.”
And so I think that experiencing the “disasters” [you fear] is actually a huge confidence builder, because you basically get comfortable with how that’s going to go. And until then, it’s a specter. It’s fear of the unknown. But once you have been through that, and you know, “Oh, we know how to fix that. It’ll be hard, but we’re all smart people, and we’ll fix it.” If you’re constantly trying not to fail, then you’re never going to do great things. You’re never going to push, because pushing often looks like failure.
I think that experiencing the “disasters” [you fear] is actually a huge confidence builder, because you basically get comfortable with how that’s going to go.
How has tooling or automation changed your relationship to your code in terms of confidence?
It’s changed everything. I can build a wrapper that tests the thing that I want to prove and then I know for sure that it’s right. Previously I would have built it into a large complex system and then tried to find a large complex way to manually do the thing and create all the scenarios that would lead to this piece of code getting validated.
Isolating that to just the thing that I’m working on so that keystroke by keystroke, I can find out whether it’s right or not. Automated testing is huge in terms of just being confident in what I have built.
Rapid deploy gives me the confidence to recover from something. So if I have a mistake, or if something does get into a production environment, either because we shipped some bad code or a customer is using it in a way that hasn’t been used before and we identify an issue, the ability to recover from that very quickly– because I know I can just fix it and push the fix and get that into production environment really quickly is– well, it’s a game changer, right?
When we started Copious, Jim [Rose] said, “We should do continuous deployment.” I immediately said, “That sounds like someone who doesn’t have to fix the deploy when it’s broken. No, this is a terrible idea. Deploys are scary and we’re going to be up all night and it’s super high risk and I don’t want to do that constantly, I want to sleep.” And then I did some reading. And it was actually in that moment that I started thinking about risk profiles, risk mitigation and understanding the risk of my action, and that by combining together a lot of work, I was creating a larger risk and that was the driver of all of the challenges of deploying. And by slicing that into thin little bits, I was actually reducing my risk profile and isolating that one little error, it’s going to be a small error and we will understand it.
I came back the next morning and said, “This is what we’re doing. We’re doing continuous deployment.” And we started building the systems to do it and never looked back.
Any final thoughts on confidence?
Well, the other thing is building something and putting it in front of your customers in the same day… it creates … I want to say it creates swagger? Suddenly I’m someone who gets stuff done. And I feel good about it and I know I have the competence because I know that it’s tested and all these other things.
The feeling of progress is fulfilling and gives you the feeling of “I’m good at what I do. I’m fixing things for customers” or “I’m making things that customers are using and loving.” And it’s a very different kind of confidence I think. But it’s a very empowering kind of confidence. I just feel good about myself and about the things that I can achieve and that makes me want to do more. It makes me want to be productive and it’s exciting. And that papers over a lot of problems. When you feel like you can just get stuff done and you’re delivering value and you’re connected to the mission.
And so I think that was probably the point where I never looked back from continuous deployment. It was just like, I never want to have to wait to put stuff into the hands of customers anymore because that’s just, that’s painful.
What advice would you give your younger self?
I would say that the learning is in the failure. What’s interesting about that question, is I think people would want to go back and save themselves from their mistakes, but if I hadn’t made all the mistakes that I’ve made in my career, I don’t know that I would be able to do my job right now. The failures are important but the key thing about failures is reflecting on them, not dwelling on them but reflecting on them because in every failure you’re going to learn something.
Yeah, that’s a weird one. I guess just: enjoy the ride because it’s going to be messy and it’s going to be gnarly but that’s how you learn.
Sign up to try CircleCI and start building confidence in your code
