Systems and Flow with Elisabeth Hendrickson
Jun 11th, 2021 | 42 minutes
Elisabeth Hendrickson is an experienced technology executive with a proven track record for growing leadership teams and shaping cultures to build leading edge products at a sustainable pace. She has led teams of all sizes, founded companies as an entrepreneur, and been an executive at a publicly-traded technology company. She's a sought-after expert, especially in the area of software process and quality. Her latest venture is Curious Duck Digital Laboratory, LLC where she is developing a simulation to give technologists insight into the non-linear nature of software development.
Rob Zuber is a 20-year veteran of software startups, a four-time founder, and three-time CTO. Since joining CircleCI, Rob has seen the company through its Series F funding and delivered on product innovation at scale while leading a team of 300+ engineers who are distributed around the globe.
Rob celebrates CircleCI's 10 year anniversary with the company's longest-tenured engineer, Gordon Syme, and one of our newest employees, JP LeBlanc, to discuss the company's past, present, and future.
Charity Majors discusses creating Honeycomb, business building practices, and the importance of proper CI/CD and monitoring. Giving us the latest insights on observability and the necessities for engineering team success.
Rob Zuber: Hello, and welcome to the Confident Commit, the podcast for anyone who wants to join the conversation on how to deliver software better and faster. Listening to episode four. Today, I’m talking to Elisabeth Hendrickson about delivery and flow. I’m your host, Rob Zuber, CTO CircleCI, industry leader for all things CI/CD. Hello, Elisabeth. Thanks for joining me. Welcome to the show.
Elisabeth Hendrickson: Thank you so much. I’m so happy to be here.
Rob Zuber: So as we were preparing, I was trying to actually describe so that we could do an introduction, how we met, and I realized I don’t a 100% know, but I’ll just say, you were pointed out to me by multiple different people over time saying, here’s a person who has excellent thoughts on software delivery, and that feels like a good enough reason. We’ve had a chance to speak a couple of times, and I’ve always appreciated your insights. So I’m really excited to have you here and share those more broadly. So I’m going to jump right into what you’re doing now, and I think we’re going to backtrack out of that because I just think it’s so interesting and cool. And so it’s called Curious Duck. First, I need to understand the name and how you got to that, but then can you give just a brief description of how you got to this and the problem you’re trying to solve?
Elisabeth Hendrickson: Yeah, totally. So it’s actually Curious Duck Digital Laboratory, so massively huge name. At the time that I was choosing a name, I thought I would probably be doing something. This was around March of 2020. So I actually filed all the paperwork to incorporate. In March of 2020, stuff ended up happening, not the least of which was this general pandemic thing.
Rob Zuber: Yeah. A few things happened in March of 2020. That’s right.
Elisabeth Hendrickson: Yeah. Yeah. And then other things happened and I ended up, I was actually going to close it all down, but then some other things happened and I ended up deciding, no, I’m going to do a thing. Now when I was naming it, I wasn’t actually sure what it was going to be. I just knew I was going to do a thing and I would be figuring out what that was. And so I wanted to choose a name that was sufficiently general, that it could cover a range of possible things and yet, also conveyed that it was in the software space. So hence Digital Laboratory, that part is the easy part. The Curious Duck, though, that has its roots. I had pages and pages and pages of possible names, trying to come up with something that was, I wanted something that was quirky and whimsical, but not stupid.
Elisabeth Hendrickson: Anyway, we have a family story in our family from when our younger daughter was teeny-tiny, she was maybe four, and she piped up from the back of the car seat on a very, very long car ride. She piped up and she said, “There’s always a duck.” And I mean, she was four, where did this come from? We have no idea. So this is a family story, there’s always a duck. It turns out she was right. I have now had the opportunity to go around the world. And every single place that I have visited from Hungary to China to India, I have found ducks. So the ubiquity of waterfowl never ceases to amaze me.
Elisabeth Hendrickson: And I use this Always a Duck in the name of a self-published collection of my blog posts that I created a while back. And so I kept coming back to duck, well, what about duck? Duck Digital Laboratory. And I after many, many words and many, many lists decided that curiosity is one of the things that I value most, and this is a Curious Duck. So that is how I got to the name.
Rob Zuber: Love it. Love it. And I appreciate, I mean, ducks are actually prevalent in software also from the whole rubber ducking concept, which we talk about all the time. I thought that was maybe where you’re going. And totally agree on the curiosity. So, okay. So now, now you’ve chosen something to do, just tell me a little bit about that.
Elisabeth Hendrickson: Yeah. So now time has passed and I chose something to do. I am in the middle of, this is actually a wild and audacious thing on my part to think that I will be able to do this, building a simulation of software development. My intention is that this is a game that people will play it and think of like the game, the Sims, sort of open-ended, but it models life. And I’ve always loved games like that, like Civilization, the Sims games, where you build up stuff over time in a system that has rules and there are repercussions to choices that you make and you have to make trade-offs. And I wanted something like that for software development.
Elisabeth Hendrickson: And this particularly came out of an experience that relates to flow. And maybe we’ll get into the example a little bit more, but in brief, the example was we were struggling to ship software and the cause was not obvious. The apparent cause to a number of people looked like it was a team that just wasn’t delivering particularly well. Spoiler, it wasn’t. It was a bottleneck within the overall system for delivering software that was actually not at all under the control of the team that looked like they weren’t delivering. But they were essentially the victim of this bottleneck. And so being able to reason about software delivery systems and see the effects of decisions in a microcosm, that’s what I want. And so that is what I am in the middle of building.
Rob Zuber: So I love it. I love the story too. And it’s a fascinating, you must have some insight into this. Why is it so hard for us to think about the system of software delivery? Like sort of as you described, there’s a simple-ish set of rules or parameters, but when they combine into a system, we struggle to reason about it and often end up at the wrong problem that we’re trying to solve.
Elisabeth Hendrickson: It is so not linear. And that starts with the fact that what we’re doing, that if you’re building a house, that house exists within the context of physics, walls fall down because gravity. So there’s physics, and there are rules there, and you can test your assumptions about the rules and we can learn more about the rules, but the rules do not change. Software exists in the world of electrons, and the rules change all the time because we’re the ones making up the rules. We, our organizations are building software that in some way captures the business rules or the whatever logic it is, it’s got behavior that we’re coding, but then we’re depending on libraries and operating systems and whatever else that give us other things. And other people made up those rules and those rules can change when you update a dependency or you update an operating system or whatever.
Elisabeth Hendrickson: So the rules can change. And that would be like if you’re building a house and in the middle of building a house, universe got an update and now gravity has a different acceleration. So all of those stress like calculations you did, no, all bets are off, your walls might fall down. But that doesn’t happen in the world of atoms. It does happen in the world of electrons. Now, take that and add on top of it the fact that you have to build anything substantial, you have multiple people, individuals and teams are all collaborating together to build this thing. And how we decide to structure that work is going to affect the way that that work ultimately gets delivered and the delays and all of the assumptions that we make between us.
Elisabeth Hendrickson: And some of my favorite stories in software development are where team A assumed one thing and team B assumed something else. And was it one of the Mars lender stories or something that was a different units? So the same number, but different units. Here, two teams made different assumptions. So you add that in, and now the world, rankly, I’m astonished we ever shipped anything. How does anything ever work? It’s kind of amazing that we’re able to deliver anything.
Rob Zuber: A bit of tenacity and a bit of luck, I think, and maybe we ship something that approximates what we started out looking for. Yeah, I think you’re absolutely right. I mean, there’s so many layers to the system. And one of those, many of those layers are people, whether it’s the stakeholders or the customer, the end user, and then the people within the process trying to build things. And to your point, we often come to this conclusion, this team is failing to deliver because this team is bad. And I think you don’t spend long in management or leadership or literally a job before you realize that’s not the only parameter. And it’s a complex and intricate system. So as I said, I love the notion of trying to help people understand that better.
Rob Zuber: One of the things that I’m really intrigued by, and we keep talking about flow, there’s a pretty famous book, which I’m going to get the title wrong… Oh, look, it’s there. It’s right on my bookshelf, Principles of Product Development Flow, which is sort of notoriously a tough read, I would say, but rich in content in detail. What is it that led you to think about this from the perspective of simulation of a game of, like, what is the difference in what you think people will take away from that relative to sitting down and reading a book on the subject?
Elisabeth Hendrickson: Okay. I’m going to take those as actually two separate questions and start with the book question. So the thing about books, I love books. Books are great. And at the same time, you can argue with a book in your head, but you can’t experience it. And so the learning by doing until you actually try to do a thing, you don’t know whether or not you understood that principle. And so for me, the thing about simulations is that the learning cycles can be that much faster, you can try and fail and try again and apply what you learned to the next iteration, and then the next challenge, the next level. So that’s why I think that a simulation is a super valuable learning tool and really important. It doesn’t replace books, but it certainly can augment books. You also asked, I think, about where did I get the idea of a simulation? And that is a different story. So do you want to go there next?
Rob Zuber: Yes, absolutely.
Elisabeth Hendrickson: Okay. I am a student of Jerry Weinberg’s. Sadly, Jerry passed away sometime ago, but I spent about 10 years learning from him. And when I went to his Problem Solving Leadership class back in 1998, I think, I might have gotten the year wrong. It might be ‘97. Anyway, long time ago, many, many, many years ago. And it was the first time I had ever been to a class like that. It was a fully immersive class, zero PowerPoint, chairs were arranged in a circle. It’s the kind of thing that we honestly, we do a lot more now, but I had never done anything like that. And in that class, it was a five day, very intense, immersive experience, there was a full day simulation, the verse works simulation. And it absolutely blew my mind because we were a little company and we had to go get customers.
Elisabeth Hendrickson: And I remember being in the middle of this simulation and having this gigantic, oh, that’s why moment, where I was acting as the president of our little company. And I was at a meeting with one of our customers and marketing. And I suddenly a way in which we had talked past each other and that I had been in that situation before and I had not, because it had happened over months, had not put it all together in the same way that I did in that moment, sitting in that little simulated meeting that lasted five minutes, that was a stand in for something that would normally be like an hour or two hour meeting after many, many pre-meetings and whatever. Oh, that’s why. And so that is what inspired me to start using simulations and games when I was teaching my own classes back when I had quality tree software from 2001 to 2012 or 2000 to 2000, whatever. Anyway, games and simulations were a huge part of what I did back then, but it all goes back to Jerry Weinberg and being inspired by his verse work simulation.
Rob Zuber: That’s an awesome story. I’m trying to piece together in my head and I don’t have an answer, so hopefully you do, why the meetings, the real meetings that you were going to, you failed to see the issue, but in the simulation they did? I mean, part of what I’m thinking is kind of game day pressure, I’m focused on trying to get the deal done or whatever the scenario was versus I’m reflecting on the process. But is there something else to that? Like, is there something about the simulation that you feel really illuminated the problem?
Elisabeth Hendrickson: Time. When you compress time, now your brain cells have the opportunity to find patterns that I think are much, much harder to see when you’re doing something over the span of months. So I was sitting in that meeting, having just had this other interaction in the simulation that if this were real world thing, probably happened three months ago. And honestly, I can’t remember what I had for breakfast yesterday. So time is the thing that in a simulated setting, now you’re experiencing all of this in a very compressed time span in a microcosm that it’s kind of like taking all of reality and throwing the most essential bits into bass relief. And so that is what allowed me to put the two things together that I had that aha about in the moment about, oh, now I see this pattern of we’re talking past each other and I’ve seen this before, but I didn’t recognize it for what it was. It was the compression of time.
Rob Zuber: Got it. That’s so interesting. So I am, I guess speaking of key takeaways, I had a chance to read some of your posts on Curious Duck about the process you’ve been going through to build this. And one of the things that I think I, well, that I took away. So I guess it’s my interpretation and I’m interested whether this is right, is that in the process of effectively trying to codify, codify, codified, whatever the rules of the game, you were forced to think more deeply about the real rules, I guess, of the world, like, how does the software says? And what is the feedback of having given something to the test team versus the, there’s a story of runaway ticket creation, I guess. And every time the developer fixes something, that creates new bugs, which is sometimes true, but I think in a good team, hopefully you’re converging on release. So has there been something really new or novel or an aha moment in terms of trying to create those rules that you said, “Wow, I’ve never actually thought about that piece of the software delivery process, but it seems it must be true?”
Elisabeth Hendrickson: Yeah. I think there have been many. I wrote about one of them recently in realizing, and it’s not even necessarily that I didn’t know this, but I certainly wouldn’t have been able to explain it to somebody that the external feedback loop is giving us information that is orthogonal to the internal feedback loop, because the way that we experience the thing that we’re delivering is inherently different.
Elisabeth Hendrickson: And sure, if you do a whole lot of work to get your internal set of intentions aligned with what your customers need, you’re really working towards product market fit, you’re a customer-obsessed, then you can get that alignment closer, but it’s still there isn’t necessarily a correlation between the information that internal feedback loops, whether that’s testers or product managers or whoever, and the external, what customers actually care about. And yeah, I knew that, but this really, again, that’s kind of the throwing it up into bass relief, realizing that, oh, literally mathematically, there will be no connection in this simulation, in the code was kind of a oh moment for me.
Elisabeth Hendrickson: And then another one just today. So today, the part of the engine my friend, Davis, and I were working on this morning, the part of the engine where the customer experiences the deployment, that’s the thing that you’re building, and they could stumble across a bug. And so the mental model now that we have that’s going to turn into code in the engine about how a bug is represented and realizing that the cost to fix the bug is completely orthogonal to the impact of the bug.
Elisabeth Hendrickson: So yes, if you would ask me that question that way three weeks ago, I probably would’ve said, “Yeah, I guess that makes sense.” But now I’ve got it really honed in my head in terms of a mathematical logical model of what that looks like. And that there are implications there, there are some kinds of bugs where if the surface area of that bug is so small and the cost is so big, AB, you certainly don’t prioritize it high, and you might never get around to that. And of course, these are the decisions that we all make all the time. But at the same time, now I can actually see that in that clarifying way that a microcosm gives us of oh, no, that totally makes sense. Okay.
Rob Zuber: Yeah. Wow. That’s really fascinating. Yeah. I think you saw my reaction was like, yeah. Okay. But if someone asked me the question without the parameters, I probably would have had a different answer based on some recency bias, whatever I had experienced that day or whatever. As you said, sort of mathematically provable, I’m picturing basically academic papers at the end of this theorems about flow and how systems work that some of them may be provable in an interesting way that we could then use as a foundation to improve all of our experiences or approaches. It does feel like you’re in an area that there are writings we just talked about, but they’re not that deep or broad, and we’re making a lot of it up and we’re not necessarily sharing in an effective way. It’s one of the things we talk about a lot more at the code level or the process level. We see people doing things in such different ways and think, how do we expose this so people understand and can share and learn from each other? So I think that’s really cool.
Rob Zuber: One of the things that, also speaking of things as I was reading your blog that was news to me or that I hadn’t considered was the, I’ll just say the tabletop gaming prototype. Is that an accurate representation? There’s talk of dice and paper, and then I thought, are you making a box to game that I can buy and roll the dice? As someone who has enjoyed many tabletop role-playing games as a child or whatever, that was very cool to me. Is that just because it’s easy? Like, what is the interest in doing it or taking that approach? What got you there?
Elisabeth Hendrickson: Yeah, the tabletop box game is an interesting idea, family fun for everybody, roll the dice, move your software development flow. I’m struggling to picture how exactly I would pull that one off and still make money. I know, but you’re right. It’s much faster to iterate on paper. And the thing that I’m building does need to be fun. It’s not enough for it to just be a simulation engine. I want it to be actually fun to play with. And that means that you have to have all those elements of fun. So again, my friend, Davis, and I have been going back and forth on a lot of this. It’s been really fun to partner with him on some of this. And we had reached a point where we knew that what we were building was no fun. We weren’t having fun anymore building it, but it has to be fun.
Elisabeth Hendrickson: And so getting together in real space, first of all, thank goodness for vaccines. I’m so grateful that that was even feasible, but it was so much faster in the space of like three hours sitting at his kitchen table. And we got his wife to play test what we had built, which still, it’s not exactly the thing that you want to put into a box set, look here, look, it’s not that much fun, but it certainly was better than, oh, wow. This is awful. So it was much faster to iterate that way. And that moving the mode got us into a different design space.
Rob Zuber: Got it. Cool. Well, I mean, as I said, maybe my kids and I would enjoy it. We might be different than enough of the audience out there looking for Saturday evening games that the box might not be the best idea.
Elisabeth Hendrickson: Actually, no, if you want that, though, so I did develop a game called Shortcut that’s a kind of like Chutes and Ladders, but not. And then the SEI took that and adapted that into a game called Hard Choices that they have now published on their site. So if you google Hard Choices, there’s, you can print out the board and the little pieces and follow along with the rules. And you too can play the software delivery game. It’s fun.
Rob Zuber: Oh, I think my kids are not aware of what’s about to happen with their summer. It’ll be great. It’ll be great. I’ll give you some feedback. Cool. So I talked a little bit about things you’ve discovered in this process, but if I understand correctly, a lot of this was inspired by how much work you’ve done in the past helping teams try to understand this. So is there anything that you brought from that experience that you thought was really critical to bake into the simulation to make sure that people, and either because you knew people didn’t have a good grasp of it or you just thought it was really core to understanding the system?
Elisabeth Hendrickson: Yeah. Well, delays in feedback cycles, one of my design goals was to make sure that people playing this game would be able to see the effects of delays in feedback cycles and how big batches, where it takes so much longer to get the feedback are so much riskier. That was one of my design intentions.
Elisabeth Hendrickson: I will say that the experience that I mentioned before about finding that there was a constraint in the system, a bottleneck, that really actually is the thing that got me rolling on this to begin with because I kept thinking there was this situation wouldn’t have been awesome at the time if I had had a little world in which I could model everything all up and then watch things flow and go see right there, because I didn’t spot it right away for sure and nobody else did either. It took so many sessions at whiteboards of, help me understand again, how do we ship software? Now, walk me through that CI pipeline one more time. What are the time measurements on these things? What is this latency here? And why do we go red? And what contributes to that? And what does it mean for the overall cycle time when we go red?
Elisabeth Hendrickson: Having those conversations over and over and over again, and then at the end of all of that discovering, oh, our tests are flaky. So they fail. So we have to rerun them, but every time we rerun them, we have to get a lock on a very limited resource, a particular type of environment, you can’t spin up a VM in this environment unless you have acquired this lock.
Elisabeth Hendrickson: So we have to acquire the lock, but that lock represents a bottleneck in the system. And the more we try to shove through the system, and the faster we go so the tests are flakier and we’re not taking the time to fix the tests because we’re under pressure to ship. And so that means that we’re shipping more less and less reliable stuff faster and faster and trying to shove it through a hole about this big. No matter how much pressure you put over here, we’re not shipping any faster. Goldratt told us that. He has this a very clear theory of application of theory of constraints.
Elisabeth Hendrickson: And so you asked about what I’m bringing into this simulation, I wanted to model that and be able to see it in a microcosm. I’m going to confess, I’m actually not there yet, because it sounded so conceptually simple. And I thought I have a few 100 lines of code, I should have the ability to on a command line, just model this. And every time I turned around, it was, oh, well, and there’s also this other concept that isn’t in this thing. And so I might’ve been able to hard-code that one scenario, but there are so many that are like that, right? Not just the, here’s this one very limited resource, but also here are these five teams that are all trying to merge changes into a thing. That’s another scenario. So I guess the short answer to your question is it’s all the scenarios, all of them, I want to be able to model all of them.
Rob Zuber: Yeah. And that’s, I mean, you obviously have a rich background and experience in this with many different teams and seeing those scenarios, which is probably a double-edged sword, like being trying to express all of that and finding a generic language. But I think that it speaks to the complexity of this system, right? Call it a system, it’s different, obviously, in every organization, every environment, but the complex number of inputs and outputs and how easy it is to end up looking in the wrong place or chasing down the wrong problem if you don’t take a step back and try to think about it from a system perspective. So it’s very, very cool to think about and a little intimidating to think about as well, I guess I would say, like trying to imagine, again, representing all of that.
Rob Zuber: So as probably a super loaded question, I’m trying to think about how to structure it appropriately, but it feels to me that this is a both critical element of understanding for people who are trying to deliver software effectively. We’re obviously, at CircleCI, we think a lot about CI and CD, which was something you can think we spent a lot of time talking about flaky tests and a lot, thank you for supporting, investing and fixing your flaky tests, but this is appreciated by all of us. How big of a gap do you think this is? I mean, I don’t know how you measure that, but as you think about the teams that you’ve worked with collectively, how good is the understanding of that system? Do people really understand the inputs and outputs? Are they working with fairly fundamental building blocks and simplistic views that lead to this kind of, I don’t know, just misdirection about what to work on?
Elisabeth Hendrickson: Yeah. I think it varies a lot. I think there are people who are super aware of every aspect of the system. I think at the most simplistic level, inputs and outputs, we used to call it requirements and deliverables, but I think pretty much everybody who has spent any time in software at all gets that level of input and output. But now you look at the system as a whole, and I have empathy for and yet am frustrated with every product manager I have ever worked with who expressed tremendous frustration, that engineering isn’t going fast enough. And therefore, we must not be competent because what they want is conceptually simple. And they have no idea what it takes to deliver it. So they understand the input and the output that they want, but what they don’t understand is everything that happens in the middle that makes it possible for them to get that.
Elisabeth Hendrickson: And so I think that that gap, it varies, but I think in some places, it’s really huge. And the upshot of that huge gap is a tremendous amount of stress and pain, and it affects people and their lives and their ability to lead a balanced life without overstressing. So that’s the other thing that just gets to me is how many people find themselves in enormously stressful and painful situations at work because folks don’t really understand the implications of the decisions. The kinds of things where a leader at some point decided not to make an investment and then they blame the team for not being able to deliver something that they needed that investment for, whether that was additional environments.
Elisabeth Hendrickson: I remember once I was consulting for an organization and my finding at the end of having spent some time with all of the various teams was one of your key bottlenecks is that you don’t have enough environments to test in. And this was different organization than one I described before. It turns out environments and environment management, ongoing common problem for most organizations. So I presented that finding to the VP who was my sponsor, and he just sort of went, “No. Look, stop talking to me about environments. What else do you got?” He didn’t want to hear it.
Rob Zuber: I was just going to ask that actually, as you were talking about that, I think systems are complex. The components of systems are simple. So diagnosing a complex system and then identifying a simple problem, I would expect that reaction. I was going to ask how many people say, it can’t be that, if it was just that, we would have known and we would have solved it? But it’s pointing out that that small problem is impacting this big system, right?
Rob Zuber: And I mean, even as an individual developer, if I’m trying to push a change out and someone else’s flaky test is in my way, I hit retry, a passes and I go back to work. But somebody somewhere is saying, we’re using twice as much time as we need, we’ve purchased twice as much hardware, or we’re spending twice as much on our amazing CI cloud provider because our tests are bad. And even that, it’s feels simple, but really, the impact on the system that you were describing is massive. But that answer, just you know what? Take a couple of days and fix these flaky tests. I would imagine people don’t want to hear that simple answer. Am I right?
Elisabeth Hendrickson: I think you’re right. A lot of people don’t want to hear the simple answer. I think that to some extent, it’s because we are choosing to optimize for one thing without fully realizing that it’s at the expense of this other thing. So if we’re trying to optimize for delivery time and are doing so by pressuring teams to not spend any time investing on anything that is not directly related to delivery of whatever it is that we’re trying to deliver, then we think we’re optimizing for time, but it’s at the expense of risk. And risk is one of those things that’s invisible. And risk ultimately translates into more time, but we don’t fully understand that in the moment that we, I say, we, whoever it is that’s making the decisions, they don’t want to hear about that simple thing, because it’s not part of their mental model for optimizing for the thing that they think that they’re trying to deliver for. This is another aspect of it. It’s just, it’s not linear. It’s so hard to reason about.
Rob Zuber: Well, and I think it’s, there’s things that come up for me. And that one, it’s hard to reason about just as an individual and then as a result, if you, I mean, speaking of collections and systems, now you have a system of people trying to communicate about issues that they’re not even that good reasoning about, let alone talking about. And so who’s going to go fix the problems that no one can understand and no one can talk about it? Even if we do talk about it, we’re talking about different things, because we have different mental models of the system.
Rob Zuber: I think there’s, one of my favorite classes of book, I guess I would say an archetype, is the type of book that introduces a set of patterns and a set of names for those patterns. And I think the first book that anyone ever gave me when I got into software development was a Gang of Four Design Patterns. I still talk about it because it allowed us to name things and talk about them easily and know we were talking about the same thing. Very clear definition, right? And I think, I don’t know if it exists, but I think a sort of clear set of patterns and nomenclature around these types of challenges and the patterns that we see that cause breakdowns, maybe it exists and people to start reading it enough or interpreting enough. Maybe it will come out of this, out of getting a fun game to play, being able to talk about it more easily solving that communication problem would drive us in the direction of maybe solving some of these problems more readily and being able to focus on them.
Elisabeth Hendrickson: I think there are some great resources out there. So Nicole Forsgren and all wrote Accelerate, fabulous book. I love the Phoenix Project and the Unicorn Project. So I think that there are some great resources that really start to encapsulate, not just the pattern language, but also key metrics that you can be looking at that are going to help you reason about this. I do think that we’re only just barely scratching the surface of where we could get if we had a shared vocabulary and a shared mental model.
Elisabeth Hendrickson: By the way, I also want to come back and say about the model that I’m building. One of the other big surprises is how small it actually turns out to be. So the core engine is currently 600 lines of Ruby and it has almost all of the key primitives necessary to really express just about any situation. It’s not a game yet, but it’s got all of these core concepts like around work and delivery and, and, and, all of those things can be expressed in 600 lines of Ruby. And when we add in now the additional stuff that I’m adding in now for the way that the customer or user experiences the deployment, that all of that might add maybe another 200 lines of code when all is said and done. So it’s actually very compact, but it does require us to really get crisp about definitions. And I think that’s the thing that’s so hard.
Rob Zuber: Yeah. I think that was the thing that, again, as I was reading through some of the stuff that you’ve published publicly, which anyone listening to this should read, it’s amazing, the kind of picture of, there’s actually a simple set of rules that define what can ultimately become a very, very complex system, I think, it’s a really cool concept to think about in the context of this, in the context of many things that we try to approach in our own software and other places. And I will certainly say, I always appreciate software that can be constructed with less surface area, less surface area for bugs, less surface area for issues, easier to understand and reason about in terms of maintenance and all those other things. So that’s a positive side effect I think of what you’re describing.
Rob Zuber: So I want to transition to one other thing that it’s kind of come up a little bit as we’ve talked about this, I want to get your just high-level read on this is, one of the areas that I see a lot of people talking about, we spend time talking about these days is optimizing for teams versus individuals. And we’ve talked about this in terms of what is it in the system? It is a system everything’s interconnected. And so as you think about, I mean, yes, there’s always some challenge somewhere, but we want our teams to be able to know how they’re doing and be successful and remove these roadblocks. How do you think about, I guess, building that focus with the groups that you work with as in team focus and then helping teams understand how they are or are not being successful? I mean, how would I know if I’m doing well at this and operating effectively as a team? And what does good look like when I’m trying to deliver?
Elisabeth Hendrickson: Totally. So the team has to be the agent of work. There are many reasons for this, but it’s a key tenant for me in leading any organization. If the team is the agent of work, then what that means is the individuals, they absolutely contribute to it. And certainly individuals deserve to have growth paths and career paths and be rewarded for their contributions, et cetera. So individuals matter so, so very much, however, individuals, if somebody decides to go on that four week vacation to Bali, that they now can, because, actually, I have not checked travel conditions to Bali specifically, but you get my point. Now people can actually take vacation, work cannot stop on the whatever it was. They can’t be the single point of failure. And so if you have the team is the agent of work, the team can swarm on things, the team can have a set of working agreements internally for how they’re going to accomplish things. There is plenty of space for the individual, but that is a key tenant.
Elisabeth Hendrickson: So then the next question that you asked was how does a team know that they’re doing well? And I think that the proof is in the pudding, that getting to that point where you actually can deliver value is absolutely essential. And so it can feel like we’re going really, really fast, we’re doing all this work, and look, we’ve delivered all these things. They’re on our feature branch. We haven’t merged them, but look, we’ve delivered all of these things. Isn’t this great? No, because nobody can get any value from that. That value is locked up. It’s locked up in this feature branch and there is more work to do before we can actually ship it. And so going all the way back to, I think it’s Ron Jeffries who talked about running tested features that you know that you’re doing well when you can point to a way in which the external world is somehow different that we have delivered real business value, we have running tested features as a result of the work that the team is delivering. That is the best measure I know of success.
Elisabeth Hendrickson: It, of course, has to be also tempered with at a sustainable pace. So continuous stream of value at a sustainable pace and while adapting to the changing needs of the business. If you have all of those components, then the team is doing well. And if the final result isn’t meeting the business’s needs, then that’s probably not the team not doing well. That’s probably something else about product market fit or strategic intent within the business.
Rob Zuber: That was like a career’s worth of knowledge in one paragraph. I don’t even know if I can follow that up with another question. I’m going to throw one thing out, because something that occurred to me, as you were saying that kind of to tie this whole thing together and maybe put a nice bow on it. You talked about single points of failure and you talked about being able to take that four week vacation. And I think we get sucked into this bus factor notion, like we have to make sure that more people know how to do these things just in case, like knock on wood. I don’t know a ton of people that have been hit by buses, surprised because I know one or two, but they’ve all come out fine, but bottlenecks, right?
Rob Zuber: Then the day-to-day, it’s actually, to me a little bit less, like you should absolutely take the vacation, but less that one time a year that you go on, ideally, it’s only, maybe I don’t know, whatever. Then you go on a big, long vacation and someone else needs to pick up the work and more, how do we balance work in our team, right? How can we work as a unit if every person just has their own little bit of the thing that they understand and we don’t have any part of that to work on right now? So you sit still while we wait for this other person to do their part. And that I think is, when you talk about system bottlenecks and it’s the testing environment, it’s the whatever, but it’s the single point of failure from a person perspective who’s just, there’s one person who can work on a thing. So I guess we can’t get any of that done. That doesn’t feel like a great. And I think it’s a really interesting combination of those concepts.
Rob Zuber: Cool. Well, thank you so much for taking the time to join me here today. Elisabeth has been awesome. I knew it was going to be awesome. The things that you think about and the way that you think about them, I always learned something. So thanks again for joining and thanks to everybody else who tuned in. If you’re enjoying this podcast, subscribe. If you want to hear something new or you want to have us talk to someone in particular, hit us up on Twitter @CircleCI. Again, thank you, Elisabeth, so much for joining me today.
Elisabeth Hendrickson: Thank you so much. This was so much fun.
Rob Zuber: Right on. Pleasure as always to speak. Take care, everybody.