Taming infrastructure with HashiCorp's Armon Dadgar
May 28th, 2021 | 37 minutes
Armon has a passion for security and distributed systems and their application to real-world problems. As a co-founder and CTO of HashiCorp, he brings both those interests into the world of DevOps tooling. As a former practitioner and proponent of open source software, he has helped design and implement HashiCorp products to solve end-user problems that he’s experienced firsthand.
Rob Zuber is a 20-year veteran of software startups, a four-time founder, and three-time CTO. Since joining CircleCI, Rob has seen the company through its Series F funding and delivered on product innovation at scale while leading a team of 300+ engineers who are distributed around the globe.
Rob celebrates CircleCI's 10 year anniversary with the company's longest-tenured engineer, Gordon Syme, and one of our newest employees, JP LeBlanc, to discuss the company's past, present, and future.
Charity Majors discusses creating Honeycomb, business building practices, and the importance of proper CI/CD and monitoring. Giving us the latest insights on observability and the necessities for engineering team success.
Rob: Hello, and welcome to The Confident Commit, the podcast for anyone who wants to join the conversation on how to deliver software better and faster.
You’re listening to episode three, taming infrastructure with HashiCorp’s, Armon Dadgar. I’m your host, Rob Zuber CTO at CircleCI, the industry leader for all things CICT. Today we’re talking with Armon about everything to do with helping developers get their software into a production environment. Welcome our Armon. Great to have you. Thanks for joining me.
Armon: Yeah, my pleasure. Thanks so much for hosting Rob.
Rob: So let’s go back a little bit and I’ve personally been following HashiCorp for a pretty long time, I think from the early days of Vagrant to start, but also there was a point early at CircleCI, where I was trying to figure out how to stop mashing buttons in AWS, and maybe write something down that described what I wanted AWS to look like, so we could manage it and I don’t know if I’ve ever told you this, but I started writing something because I couldn’t find anything, but it turned out there was maybe some SEO work that hadn’t quite been nailed yet and like Terraform 0.1. But that was, I think, the beginning of a long trail of pretty impressive products that HashiCorp has put together to help us all be better at managing infrastructure and helping get software delivered effectively into our infrastructure environment. So what was the inspiration for you and Mitchell to really start HashiCorp in the first place? What was the problem that you were really aiming to solve?
Armon: Yeah. So in some sense it goes, it will still take us away back at this point. So, kind of fun bit of trivia history. I actually, I worked at Amazon, at AWS back in 2008. And so, I remember being there on the ground floor when they sort of showed off cloud formation. And I remember thinking to myself, I’m like, man, this is like a really kludgy syntax. Do we actually expect anyone’s going to define functions and JSON was horrible syntax and this is not very user friendly. But I think the view was like, Hey, it’s early days, this stuff’s going to evolve and people will figure it out. I think flash forward a few years, me and Mitchell end up sort of at a mobile ad tech company together.
And in some sense of it, we felt like the tooling really hadn’t evolved, right? It’s like now it’s like [Circuit 00:02:29] 2011, 2012. It’s been now call it six years since the launch of cloud and everything just still felt so raw. Right. It was just like, if you want to do provisioning, you’re kind of building your own provisioning tool. If you want to do the secret management or service discovery, or really orchestrate an app at scale, any of that stuff is like, you’re building it yourself. And I think everyone we talked to that was kind of the experience, right? When we talked to people like GitHub and say, Hey, how do you do it? Or people had Stripe, how do you do it? And everyone was like, we build it ourselves. And so I think for us, the kind of inspiration was feeling like, okay, if everyone is reinventing the wheel here, maybe someone should just like write a good provisioning tool and everyone else should use it rather than everyone rewrites the exact same set of tools over and over.
Rob: Yeah. Right on. And so that, I think Vagrant was sort of a side project initially, but then you had some inspiration around cloud and managing cloud, which I know we’re thankful for. I’m sure there are many, many other people listening to this that are thankful for that. So you start the company. But as I said, that was the beginning of a long suite or a large suite, I guess, a long list of products that have helped in the infrastructure space. Was there a particular customer that you had in mind or sort of set of problems or a way that you define the bounds and has that changed at all, as HashiCorp has evolved?
Armon: So, it’s interesting. Yeah, so to your point, Vagrant coming in was the one product, it was a predated HashiCorp, sort of a side project Mitchell started and we kind of had that coming in. So, that was sort of an assumption. I think what we sort of did, once we said, Hey, we’re going to found HashiCorp, we’re going to go to go build this, me and Mitchell basically spent kind of, I don’t know, two, three months, I’ll call it kind of notebook designing of the company. Right? So not a line of code, just working out of our notebooks in terms of like, Hey, what’s really the problem space? What should the solution look like? And I think in some sense to your point, the target user and target customer, if you will, was ourselves. And we said, okay, what would be the tools me and Mitchell wanted? Right.
In our role as an operator, in our role as someone who’s managing a building cloud infrastructure, what are the things we wish we had? Right. And so we started with that and said, okay, where would we divide the sort of the lines in the sand, I guess, in terms of, what are the kind of core problems here? And then what we ended up with was saying, okay, there’s a set of problems that are really well solved. Right. Things like, okay, there’s folks like Datadog doing observability really well. There’s folks like you guys who are focusing on CICT we’re not going to go tackle those problems. Right. GitHub’s doing version control, et cetera. And so we said, okay, anywhere where we feel like someone’s doing a good job, let’s just not touch it. And then the pieces that we felt like were like, Hey, no, one’s talking about like, how do we do packaging in an automated codified way.
Right. And so today that’s packer, how do we think about kind of provisioning and lifecycle management and sort of true infrastructure as code, not at a config manager level because there was people like Chef and Puppet at the time. They were doing it at like, how do I configure a VM, but infrastructure’s code for my whole data center, my whole infrastructure. This was like, okay, that also maybe became Terraform. And then console was like, how do we know… We have the strong view around, we want an infrastructure to be immutable, meaning, we don’t want to hard code IP addresses. We don’t want to run Chef and Puppet in production. So you need some dynamic binding, right? You need something that’s going to do service discovery for you so that, it was console. And then we needed an app deployment tool.
And so Circuit 2012, there was no containers. There was no Docker, there was no Kubernetes. So Nomad was sort of this piece that we’re like, okay, we need something that’ll decouple the application layer from the infrastructure layer. We don’t know what the format is exactly. We don’t know how exactly it will work. But I think at the time when we found it, we basically knew there’s these six pieces that were kind of the I’ll call it the V1 of the HashiCorp portfolio. It took us, I don’t know, three years, four years to actually build, to get to V1. And then basically we actually didn’t create any new products for five years after the initial portfolio. It was only last year that we actually added two new products, Boundary and Waypoint. But we were pretty focused on that initial trench for eight years.
Rob: Yeah, that’s a really long time actually. I was about to ask that and you sort of expressed it, the idea that you had a really good picture of the boundaries of that, the portfolio up until fairly recently. So was there a particular shift, you mentioned moving from Chef and Puppet to Docker and containers and sort of how we think about immutable infrastructure. A lot of that has changed over the period of time that you’ve been around. Was there a particular shift that led to those most recent additions and does that fit in what the general concept was that you had of the portfolio and are just new opportunities or was that really a shift in the market?
Armon: That’s a really good question. It was a little bit of both. And I think part of it is like, by talking about like Boundary, for example, it was like a problem that I don’t think we fully recognized when I started the company. So I think the problems still exist at eight years ago. I just think we weren’t as aware of it. Right. And I think the default view was VPN and SSH, bastion hosts and things like that. I think the trend in industry that’s really kind of accelerated last few years of kind of, how do I move to a zero trust architecture? How do I go to kind of principle of least privilege and not giving people network access to things. And so I think that trend kind of inspired us to go work on Boundary to say, okay, what would it look like if you didn’t have VPN and you didn’t have bastion hosts, how would you still do privileged access, but in a modern way based on identity, based around least privilege, kind of these industry trends.
I think that one did sort of evolve a little bit out of it. I’d say Waypoint is a funnier one, because I think in our mind’s eye, the original portfolio was very much meant to be developer oriented. Developers would use these tools and build applications and consume cloud infrastructure. I think if I’m brutally honest with ourselves where we missed the mark a little bit is certainly everything HashiCorp makes by and large is an operator tool, right? They’re powerful, they’re complicated. They’re mostly focused on operators who want that power and flexibility and configurability. They’re almost too powerful for developers. They’re almost too capable. Right. And I think, especially when you sort of put it in front of a developer and say, Hey, to do modern cloud infrastructure, you need to learn Vault and Terraform and Console and Kubernetes and the AWS data model.
And it’s almost this overwhelming experience for the average developer was like, yeah, I really don’t care about any of that, whether it’s serverless or Lambda or container or VM, like I frankly don’t care. I want to work on my app and deploy it and whatever. Right.
Armon: And so I think Waypoint was sort of us acknowledging that we sort of missed the mark, the goal we set out to solve eight years ago, it was like make life easy for developers. And what we did was make life really easy for operators, but we sort of missed… We sort of forgot about the developer along the way.
Rob: Yeah. Well, I think that’s a really interesting perspective. One of the things that I talk a lot about as I talk to folks about dev ops is this notion that developers learned a little more about operation and operation or operators learned more about development, right. And infrastructure is code and a lot of that type of tooling was really if I can project onto your story a little bit, bringing a developer perspective to what was typically an operator problem. And so certainly enabled more of a bridge between the two. But yeah, I think that a lot of developers now are like, yeah, it’s super cool that I can build a service, but I really just want to build the service. I don’t want to build the service and then understand how it’s deployed and understand how it’s networked and all of these other pieces.
We’ve kind of asked every developer in a lot of instances to understand a whole bunch of stuff that wasn’t part of their remit before. Kind of, we keep talking about reducing cognitive load, but meanwhile we’re increasing it. And so I’m actually curious how you think about that. Again, you reflected on just Docker containerization, how we deploy. I think that has enabled us to start breaking things down into more services, et cetera. But a lot of that has driven new complexity in the name of reducing complexity, right? Like, oh, you only have to worry about this one service, which is true. If all I worry about is that service. But if I understand everything about how it’s deployed and how it talks to other services, I’ve sort of taken complexity and pushed it into the system and out of my code base, does that make sense? And does that align with what you’re trying to sell for it?
Armon: I’m laughing because me and Mitchell talk about this all the time, which is, I feel like when I started doing software development, what did a deploy look like for me? It’s like, great. I would write my PHP app, then I’d use my desktop FTP client connect to my Linode or whatever. I would upload the latest version of the PHP file. Apache would reload it and it was deployed. That was it. Right. So I had my gooey app, dragging brought my file and that’s how a deployed worked. And so that’s how sophisticated… There’s a lot of problems with that. Don’t get me wrong. There’s a lot of problems. But from a user experience perspective, it was simple and I didn’t really have to think about much. I wrote my PHP, I tested it locally. It took me two seconds and I could script my FTP client.
And that was it. And now you think about to your point, it’s like I have to go write 10,000 lines of YAML and understand 50 different systems and how they integrate with each other to deploy effectively the same application that was deployed before. So I think you’re a hundred percent right, in the name of simplification, we’ve actually made the user experience tremendously complicated. And you might argue with me like, okay, but those are equivalent now it’s load balanced and scaled and highly available and blah, blah, blah. Sure. That’s all true. But the developer doesn’t care about it.
Armon: Right? Like, yeah. As an operator, I’m happy that it’s highly available and distributed an audit scale. But like as a dev, I’m like, I just want to deploy my app. The rest of this stuff is noise.
Rob: I totally agree with all of that. And one of the places that we see that going in our own customers is people are coming to that realization, Hey, this is actually great. First of all, if you’re a two person startup, because you’ve been there, maybe don’t start with that. Right? Like start with the PHP page that says, Hey, we’re thinking about doing a thing. Does anyone care? Because it’s a lot easier to get that done and learn, than to go build this highly available system. But when you get to the scale where you do derive value from that, you want to make sure your developers are still able to focus on delivering the thing they’re meant to be focusing on whatever the capability is for the end user versus understanding how the entire system is deployed. So are you seeing this sort of same, I guess, trend of developer experience, developer productivity teams, and are they landing in the core of sort of your customer base, is that who’s now looking at some of the HashiCorp products and really owning that within those organizations?
Armon: Yeah. I think you hit it right on the head, which is, I think there’s this… It’s about the sort of the right structure or the right process for the right size of the company. Right. So to your point, you have the two person startup, just go with the giant VM and do the FTP deploy, right. That’s the right answer, but I think you’re right. And I think that is what the pattern we’re seeing in sort of the larger customers, larger enterprises is in investment in sort of those core platform teams. And then on top of that, is sort of the I’ll call it the developer experience teams. So how do I wrap that underlying platform, which might still be relatively complicated because it has to deal with scale and multi data center and PCI and all these different security compliance things, but let’s wrap them in sort of a developer experience layer that tries to simplify what the internal customer really experienced.
You’re just a dev team that wants to push an app out. You don’t really care about how the plumbing works. Right. And so I think that is very much the gap. And I think the question that HashiCorp, we would get all the time from customer’s like, okay, great. We’re using Terraform and Console and Vault and these things. What’s the layer we should expose to the developer? And I think in some sense the answer, and we were always honest with customers, sort of none of those. They’re not really developer tools. You need something else. And so their question was always like, okay, but what is that something else? And I think for us that became the motivator for Waypoint was like, okay, we need an answer that’s just not, no.
Rob: Right. No, you’re doing it wrong. Go fish.
Armon: Yeah. Go figure it out.
Rob: Yeah, just try again and we’ll let you know if you’re right this time. That’s really, that’s really cool. So, for listeners that aren’t necessarily familiar with it, can you give the few sentence summary of kind of the problem as you just described were you were aiming to solve and some key tenants of Waypoint in terms of how developers should think about it?
Armon: Yeah. Yeah. I think at the high level it’s pretty, pretty straight forward, which is like our view is from a developer workflow perspective, there’s a number of these kind of, I’ll call it key life cycle steps. Right. And I think the left-hand side of sort of, if we think of it as a pipeline is really pretty well understood. Great, I’m going to probably use like VS code and develop in that. Now I’m going to commit to Github. Right. Great. That’s my version control. I’m going to run my CI using Circle to run my tests, do my integration. Great. So the left-hand side pretty well known. Now the question is how do I go to production? I think that becomes the sharp fall off of a cliff basically, which is like, I think there’s three or four key steps from it.
One is okay, how do I build a production worth the artifact? Right? And so that might be a container, might be serverless sip file. It might be an on-prem VM image, et cetera. So there’s this, okay. I might have this fragmented workflow depending on where this app needs to deploy and what tool I need to use. Then I have a challenge on, okay, how do I truly do the deploy? Right, right. I just want this thing running. Do I have to right YAML because it’s on Kubernetes. Do I have to write, it’s a Terraform to script with VMware to deploy, right, et cetera. And then there’s sort of a release orchestration. I think we’re really early in developers internalizing this, but once you’re on a more sophisticated app, it’s not just… The deploy is not the release, but I’m doing a deploy.
And then I want to do blue greens and canaries and gradual traffic shifting and things like that. So I think those become the kind of… And then at the end of that pipeline is observability, right. I need to then have an eye on is everything working? And so for us, it’s really looking at that kind of the second half of the pipeline, particularly around the kind of that build deploy release phase and saying, how do I provide almost a Heroku-esque experience there where regardless of what the plumbing looks like, great, a build is always Waypoint built. That might result in a container. It might result in a VM image. You don’t have to care as a developer, right? It’s the same workflow. Then a deploy it’s always Waypoint deploy. Maybe that lands on Lambda, maybe it’s on ECS.
Maybe it’s on Kube. Maybe it’s on a VM, you don’t care. And then release managers sort of like, Hey, what do I have to do to interact with API gateways and load balancers and traffic shaping policy to kind of do that gradual release management. And again, hide that from the developer, right? Like they care about the fact that they’re doing a release. They don’t really care. I’m using F5 or an API gateway or whatever it is. So from the developer viewpoint, it’s that Waypoint, looking at the build, deploy release phase, from the operator’s perspective, right. And I think, we stay very grounded us sort of being an ops company is saying, how do we allow the operators to kind of bring the different pieces of their infrastructure and specify how that should work, not surprisingly in an infrastructure-esque code way.
Right. Kind of consistent with everything. We’ll say, okay, here’s how the bill process should work for our container. And here’s how it should work for the VM. Here’s how it’s going to deploy on Kube. Here’s how it’s going to deploy on Lambda. Here’s how we’ll use Console to do traffic shaping to the release management. So they’ll kind of define that in some sense of below the fold, right? So I kind of define as there’s kind of this platform layer, which divides the developer concern from the ops concern. Ops people will define how it works, how the plumbing exists and links together. And then the developer doesn’t have to care, right. Their experience is cool. My test pass, I’m ready to do a deploy. Great. I just hit Waypoint up and it does the build, deploy and release. And I don’t have to think about it basically. Right. So I think that for us is kind of the core heart of it is kind of that providing that sort of Heroku-esque developer experience.
Rob: Makes total sense. And so one thing that I’m always reflecting on as I think about this, that split and where we build these dividing lines platform, for example, is we still talk about developers and operators, right? And dev ops, I guess for anyone watching on video, they could see my little air quotes around dev ops, ultimately started as a cultural shift, right? As we thought about working together as teams. And then we talked about sharing practices, operators learning more about great tools they can get from developers, developers learning more about operating effectively. And in some regards I feel like we did this horizontal tilt and I don’t really know how to describe it, but where developers are all the way out in production, carrying pagers, owning the services at their level, and then what are now infrastructure SRE.
We’ve changed the names a little bit, but folks who are operating, are operating the infrastructure below that and then creating sort of a nice interface at the platform level. And it’s almost like we’re now trying to talk less again in a way, but in this way that like, in team typologies, for example, if you’ve read that, there’s a lot of discussion about finding the points of collaboration, identifying them and minimizing them. Not because we don’t like each other, but because collaboration is expensive.
Armon: [inaudible 00:20:27].
Rob: Yeah, exactly and so I’m curious if that resonates, if that makes sense to you, because I feel like you’re talking about the same thing, which is basically, I want to create a clear boundary again, between… It felt like this aha moment where we all started talking to each other and now we’re sort of like, cool, I got what I needed from you. Now let’s go back into our corners and do our jobs effectively because we have a better toolkit to do that. Does that make sense?
Armon: Well yeah… And this gets to, maybe this is a controversial-
Rob: I hope so.
So I can do that, but is that actually an efficient way to run a company? Probably not. A) those people are hard to find and B) it’s sort of like flies in the face of sort of any form of specialization of concern. And so I think to me, it’s really about acknowledging, okay, my developer is really good at the front end and that’s their strength and that’s their passion. How do I let them focus on that rather than have to worry about publishing infrastructure, right. Because if that’s where their passion lies like, okay, great. Let me let them be really good at that and hide the other stuff from them. And so to me, it goes back to that idea of like, what are the contracts between these teams? Because there’s people who are truly passionate about the infrastructure pieces and want to make great developer experiences and enable their teams and their tool builders at heart.
Right. And that’s where their passion comes from. And so to your point, I think it’s not around avoiding talking to one another, because I think that’s sort of bad, but I think it’s one of these things, how do you create a clear contract between these teams where it’s like, okay, great. My contract is Waypoint is the interface, right? Great. As an ops team I’ll own and define the Waypoint configuration. You’ll be my downstream customer. You don’t have to worry about that. You just run away with my deploy and the rest of the sort of hidden [inaudible 00:22:52]. So it becomes the sort of point of contract, as opposed to like, let’s negotiate this or talk to one another and schedule a meeting or whatever, which becomes a high friction interaction.
Rob: Right. Right. Yeah. I think that’s exactly right and the historical point of contact or the contract, sorry, was I’m going to hand you this thing that I built that you don’t understand and if anything goes wrong from this point forward, that’s your problem.
Rob: Which was uncomfortable for all of us. Right. And now it’s more, we own this, you own that, these are the services we provide to you, but the full life cycle of this is in your hands as a developer, in my hands as an operator infrastructure maintainer. Yeah. Love it. So is there anything in the, I guess in your team, you talk about being efficient at growing companies, how you think about hiring folks and stuff. Is there anything that y’all are doing at HashiCorp in terms of how you build, how you deliver, deploy that you think is indicative of maybe some places you’re thinking of going in the future, like you’re learning from yourself and saying, you know what, this is working really well and more teams should be doing this?
Armon: You know, it’s… Yeah. The thing that pops right to the top of my head is we’re in this interesting transition as a company. Right. And I think actually you guys are kind of going through the same thing as us, which is historically HashiCorp was a desktop software company. Right. You think about Terraform, you think about Vagrant, we build and release desktop software. So we do it on… We have a quarterly ship train, right. And we plan around these 12 week dev cycles and it’s the irony and what’s not lost on this is we… Yeah, we sell and we build desktop software to help people adopt cloud where everything is a managed service. And so it’s like sometimes we feel like a little bit of like, we’re kind of like the dinosaur watching the meteor heading towards us and we’re like, huh. Yeah. So maybe we should be like a cloud service company.
So, I think we started making a bunch of investments there. Right? We announced a bunch… Terraform cloud is the first managed product for us, we announced our HashiCorp cloud platform. And so I think now we’re kind of going through this shift of all of this stuff is becoming a cloud service, but in the process of doing so, what we’re realizing is like, the way we used to build before was each of these things as an independent silo, right? Like the Terraform team by and large, doesn’t talk to them about Vault team. They’re independent teams doing their own thing. But once you talk about it in the context of a cloud service, and then the context of like, well, it all needs to be on the same HashiCorp cloud platform.
Then there is this strong investment and focus on a platform team and how that platform team then becomes sort of an enabler and gives leverage to these other product teams. And so it’s been super fun as like, as that platform has now been maturing, great. The first product took 18 months to onboard. The second product took eight months to onboard. The third product is only going to take us five months to onboard. And now we’re doing some of these things where we’re going from zero to customers are trying it in three months, because you’re able to see that acceleration and that leverage that you’re getting from having a true platform in place. And so I think to us, it’s in some sense kind of, that’s been the pattern and substance of what we’ve preached to our customers for a long time. It’s like you should have a central platform team. And sort of the disconnect has been like, we don’t necessarily do that because we’re a desktop software company. So it doesn’t make sense for us.
Armon: But now it’s been funny as we kind of eat our own dog food or live our own philosophy, however you want to put it as like yeah. That leverage you’ve get of having a strong central platform team really is kind of tremendous.
Rob: And was there… Obviously the first one is going to take a long time because it’s not even a platform, it’s just a piece of that first deployment it’s then finding those common patterns. Was there anything that was particularly surprising as you tried to get leverage where you thought, oh, we over centralized in this spot or that spot. I feel like that’s one of the bigger debates that people always have about platform teams, like the balance between autonomy and the value of kind of that independent ability to deliver versus shared capabilities that I don’t have to worry about that thing anymore. Was there anything that really surprised you in that balance?
Armon: Oh my gosh. So many things. I feel like the constant battle for us is exactly this, right, is the product teams will want to run out ahead of platform because they’ll be like, Hey, customers want this capability or this feature. The underlying platform doesn’t yet support that. So we’re going to go build it in a bespoke way. And then the platform is like, okay, but you’re going to go build whatever audit logging, let’s say, but that’s a common feature that all of the products will need. So it doesn’t make sense for you to have a bespoke thing, the platform should have it. And so there’s this sort of constant tension of do you prioritize the platform consistency and that lift, but it might be at the expense of slowing down a product team’s actual delivery.
And so I think so much of it, I think it’s kind of that organizational challenge of navigating the priorities and the timelines and the dependencies of that stuff. So I think realizing that that’s so much harder than I think we first thought about for sure has been hard, but I think as the platform matures, then it’s kind of becomes the opposite where people are like, oh, I get all of this stuff for free. Why would I even bother do it any other way? I’m obviously just going to start to here basically.
Rob: Well, you talked about onboarding to the platform in three months or less. I think when you… The first product that you… I don’t know if this is true of the last two, but the first product that you build on top of the platform and all of the work that doesn’t have to get done to get there, I think will be the… That’ll be the real next step or next next level of delivery. Super cool. There was one other area that I really wanted to talk to you about, you mentioned this a little bit as you were talking about Boundary and the zero trust and state of security in the world. And I think I don’t have to give specific examples to just say, this is becoming more and more prevalent in all of our thinking as we deliver in the cloud as we deliver software in general.
And so I’m curious if there are any interesting trends that you’re seeing now that you have Boundary out there, sort of what your customers are coming back and saying, awesome, but we really need this because this new direction is scary for us. And then on the other side, I think we always get stuck in our, I’ll say bay area bubble, being in, talking to advanced tech companies all day long, how well understood are these problems and are people really ready to solve them?
Armon: Oh yeah. I think maybe starting there, I don’t think they’re super well understood. I think we’re just now at the cusp of people, even sort of trying to acknowledge that zero trust or that we need to move away from our traditional kind of view of security. I think historically we overdependent on the notion of the network perimeter, right. We said, okay, this is… In some sense, it’s so bad that it’s baked into the language we use. When we talk about a private network, we’re just so used to it, but what does that language mean? We’re creating a distinction between our private and public network and the implication being that, oh, by virtue of it being this private network it’s secure and high trust and confidential, right. It’s inherent in the language and so I think, how do you start to unlearn that?
How do you start to be like, no, there’s no public private network. There’s just the network, right? It’s all untrusted, it’s all bad. Right. So I think part one is like, we sort of have to unlearn in some sense that the private network is a meaningful construct. That’s piece one. I think problem two is zero trust is so overloaded of a term. Right. I think you get into some conversation with people, like, okay, but what does that… Does it mean I don’t trust my operating system? Does it mean I don’t trust the hardware it’s running on? Does it mean I don’t trust the CPU? Right. And you can get to any level of paranoia, that’s the thing. Right. And there’s like, yeah, great. If you’re at the department of defense, you should probably be that level of paranoid, right? Yeah.
Those are your realistic threats. The average customer and certainly the average startup rates should your threat model be, my hardware manufacturers not to be trusted, probably not. That’s probably an extreme threat model for you. Right. So I think piece one is people really starting to internalize the idea of like, okay, what are reasonable threat models? I think for most people, starting from the point of like, my network will be compromised is probably a good starting point and then figuring out from there, okay, well, what are the downstream impacts? If I say, Hey, I’m going to start with the assumption that my network’s been breached. How would I now design my infrastructure around that? Right.
Armon: And so I think that’s the hard part. And I don’t think we’re there. Right? I think as an industry, we’re first… We’re still trying to, I think most people I talk to us are still trying to hold onto the idea of a private network.
Rob: Well, I love that framing. I feel like in a large number of conversations recently, I’ve come up with this abstract pattern that is, this outcome is inevitable. How do we plan to make it less painful? And honestly, as a totally random example, this came up, I was talking to someone about the fact that we’re constantly read… Basically constantly replacing the front end stack that we use. Right. Every few years it’s like, oh yeah, react is the hot thing everyone’s going to use react to announce angular and I’m kind of going backwards now it’s Ember. So play those the other way. And so like, could you actually build all of your front end in a way that didn’t matter what framework it was being used on? Can you isolate your business logic so that you just accepted that?
And I think that pattern is surprisingly common as we think about the way that we try to conceptualize problems, right? Like to your point, the private network, right? We just put this perimeter up and then we can be safe inside here is not a real thing. So if you assume that someone’s going to be inside the network, then what would you do differently. Right. How would you design that? And I think, for some reason, I’m sitting here looking out the window at a bunch of cars. And I think about this, people who just leave their windows down because there’s nothing in my car. And so they think about like, if I leave my windows down right, then I don’t leave valuable things in my car. I’d never run into this scenario where someone smashes my window and takes my stuff.
Right. It’s a very simple comparison. If you just… Almost what if we didn’t have a firewall? What if we didn’t block these ports? Then how would we behave so that it would actually be okay to do that. And I think to your point drives you into a much more interesting frame of thinking instead of just trying to be tighter and tighter. I’m not saying don’t have a firewall, because you might as well keep most people out, but someone’s going to get in. So what are you going to do about that fact sort of thing. And so is there, again anything particularly interesting about where you’re thinking of going with that product that is framed around that type of thinking?
Armon: Yeah. I think to me it’s like, there’s a number of implications of this thing. Right. So I think if we say, okay, I’m going to let go of my perimeter and I think to your point, you’re exactly right. It was just like, well, I’m not ever telling, people to get rid of your firewall. It’s still a valuable line of defense. It’s just don’t treat it like the last line of defense, right? Like baby with bath water type of thing. But I think then the implications is like, okay, now I actually have to care a lot about human identity, machine identity, machine to machine workflows and human to machine workloads. Right. To me that becomes the kind of matrix that we think about is like, everything really becomes an identity sort of in control.
Right. What I really stopped caring. I was like, this IP address can talk to that IP address, which I actually think is great because it’s like, to me, it’s in a cloud world, the idea of mining to manage static IP lists of like allowing denialist is like, it’s insanity, it’s right. It’s too dynamic and stuff is a femoral. I never wanted to manage that stuff to begin with, let alone in a cloud setting. So I think a nice thing is like, how do we let go and move to an identity world where I say, great, Armon, he’s a member of the DBA group. The DBAs, they’re allowed to reach my database. That’s a nice identity driven thing where the set of IPs that represent database, I don’t care what those are dynamic has got to come and go, right.
The set of my DBA is that might be dynamic too. A new [inaudible 00:34:57] person joins. They get added to the group. Okay. I don’t want to have to go update a bunch of rules. Right. So to me, it’s about get to that identity shift and then how do we actually enable that everywhere? So I think that’s where we’re spending a bunch of time sort of on that thought train is like, okay, with Boundary, great. Piece one of this is like, you need to have a notion of human identity. So, we just released a bunch of tight integration with OIDC providers. So people like Okta and active directory, like you’ll can pull in that identity. But then the next piece of it is like, well, what about the application of machine identity? You want to be integrated with Kubernetes and with Console and with the cloud API so that I don’t have to keep a static list of databases anymore.
It’s like, yeah. Boundary, go to Query Console and say, tell me the set of databases, right? Or talk to Kubernetes and say, tell me the set of web servers. And that’s the more like natural way I think that this access control should go. And like same thing when we talk about service to service. Right. And I think people get lost in the jargon of things like service mesh. And as an industry, we do people a terrible disservice because we love our buzzwords, but we don’t like it crisp articulation of the problem. To me, the crisp problem is just like, I want to say which services can talk to which other services, and then it should just be enforced at the network layer. Right. And I don’t want to deal with the IPs.
Armon: That to me is the real value of thinking about these machines and machine workflows. I can set a rule and since my web server can talk to my database, I don’t care if I have one, 10 or 50,000 web servers. That rule is the same. Just make it work.
Rob: Yeah. I think what’s really… It’s a great articulation of that and what’s really fascinating about it is that description helps with security, but also makes a lot of other things easier. And one of the things that I think I continue to come back to from a security perspective is as simple as generally more secure, right? When we have systems that we can understand, it’s a lot easier to think about how to manage the security of those systems. I think a lot of the challenges we run into are when it escapes our ability to even comprehend the full system. And then of course, there’s gaps. Right? Of course, we’ve missed things that are open. Right on. Well, thank you so much for joining today, Armon. It was super fascinating to talk about, well, literally everything that you talk about. I obviously love all these subjects. I really appreciate your perspective on them. Congratulations on the continued success of HashiCorp. You all are doing amazing, amazing things and just really appreciate you joining me today.
Armon: Yeah, my pleasure. Thanks so much. It was great to join and great to chat.
Rob: Right on. And thanks everyone for tuning in. If you enjoyed this and you want to share it, you can find us on all the classic places that you subscribe to your podcasts. And if you want to suggest anything that you want to hear about in a future episode, find us on Twitter at CircleCI. thanks again, Armon.