2019-04-02-Terraform-v6.jpg

Hi! I’m Chris. I’m a Solutions Engineer with CircleCI. I love speaking with prospective customers and technology partners, solving challenging issues, working with very talented engineers, and helping folks get the most out of our platform. My role is very much at the intersection of engineering and sales. At this intersection, I do a lot of demos that show off complicated things in deterministic ways, and I constantly find myself spending time recreating demo environments. Before transitioning into this role, I was on CircleCI’s SRE team for about two years. On the SRE side of things, I discovered a wonderful piece of software that helps me with my demos called HashiCorp Terraform. I’m sure many of you are familiar with it, but if not, here is the tl;dr:

Terraform is a magical tool that enables the automatic provisioning of infrastructure based on declarative templates, written either in HCL (HashiCorp Configuration Language) or JSON. These manifests represent declarative snapshots of what the desired state of your infrastructure would look like. Emphasis on declarative rather than procedural.

This is the first post in a two-part series. This post will focus on the basics of Infrastructure as Code (IaC), addressing a fundamental question that I’m often asked when speaking with folks who are new to the platform: “Can I use Terraform on CircleCI?” The answer is yes. Yes you can! The followup post will contain best practices for executing Terraform on CircleCI and address more sophisticated use cases.

Declarative vs procedural programming

Historically, developers and operations folks would set up elaborate procedural scripts in Bash to facilitate automation and configuration of resources. Procedural programming represents a step by step set of instructions; do x, then y, to achieve z.

The declarative programming paradigm says that if the desired state is z, then let the tooling take care of x and y automatically. We declare the desired end state and then the underlying machinery does the heavy lifting.

This new methodology for automatically provisioning servers, network infrastructure, and cloud resources, is referred to as Infrastructure as Code. IaC is a relatively new paradigm and is changing the way technologists interface with and manage cloud infrastructure. As engineers, we’re used to storing things, all-the-things, in version control. This includes resources like READMEs, artifacts, test results, configuration files, etc. Why not store declarative snapshots of the underlying infrastructure that we host our applications on? Think about it. We’re also accustomed to automating things, all-the-things, whenever possible. With modern testing frameworks, we’re able to automate the enforcement of testing standards across all of our projects; with tools like CircleCI, we’re able to automate deployments; and with tools like Terraform, we’re able to automatically provision infrastructure.

We use Terraform heavily at CircleCI. We use Terraform to deploy and maintain our complex, ever-evolving cloud infrastructure. The self-hosted installation of CircleCI is packaged via Terraform. Many of our customers use Terraform. In addition, I frequently use Terraform to set up demo environments and manage my team’s shared AWS resources.

Determinism, visibility, and automation

Let’s talk about a few reasons why you’d want to execute Terraform on CircleCI rather than a developer’s local workstation. Fundamentally, there are several reasons why we’d execute Terraform on CircleCI:

Determinism. Terraform is elegant in its simplicity. It’s a static binary written in Golang (I <3 Golang). Running it in an isolated environment with uniform input will almost invariably produce uniform output. We refer to this in computer science (and higher order mathematics) as referential transparency: same input, same output. No more “well, it worked on my machine…I’m not sure why it crashes on yours”.

Visibility. Every pipeline, every workflow, and every job is stored on CircleCI for posterity. You can look back at the job output from a job that ran 2 months ago. This lends itself to auditability and visibility. Other team members can see the output from Terraform. They can plan, look at errors when they come up, see the output for each and every run, etc. This is very powerful. We can look at every commit associated with every build. Should something go wrong, we can SSH into a job and look at the corresponding logfiles. Most importantly, we’ll have a record of each and every deployment.

Automation. Every time a developer writes and commits a new module to their VCS, a workflow is triggered, validating the plan file and setting the stage for a deployment automatically. Syntax errors and formatting issues can be captured automatically. With some branch-level filtering in place and a manual approval gating the deployment to production, you can automatically deploy to a staging or QA environment every time a successful change is made. This is just the beginning; imagine being able to include the dynamic provisioning of infrastructure as part of a series of integration tests, or being able to deploy a k8s cluster automatically.

Conclusion

The benefits of IaC are enormous. While infrastructure as code pertains to, well, infrastructure, the same kinds of “configuration drift” that invariably occurs when manually provisioning servers/services are equally applicable to networking infrastructure, subnet configuration, ingress/egress rules, load balancer configuration, etc. IaC alleviates many of those issues and more. It’s powerful, it’s elegant, and it’s how many teams and organizations are managing their resources in the cloud.

Read more: