Build cloud infrastructure from your CI pipeline with Pulumi

Modern software systems are complex, with services distributed across data centers, in many zones, all around the world. Gone are the days when we managed individual servers dedicated to our organization, comfortable with the knowledge of the unique quirks of our setup.

Now we rely on others to manage massive data centers where we borrow small slices of virtual space on shared hardware, traveling over shared networks, all in a system we call the cloud. Managing that interaction with the cloud is part of what cloud engineering is all about.

To deliver applications cleanly, you need to manage infrastructure with pipelines just like you manage continuous delivery. You can bring the practices of application delivery to infrastructure as code with the maturity of cloud engineering.

Getting started with cloud engineering using Pulumi and CircleCI

Cloud engineering brings the cloud closer to application development, applying engineering practices and principles to infrastructure and innovating and collaborating faster across the entire team. Pulumi is an infrastructure as code platform you can use to help create a cloud engineering culture in your organization. This kind of culture includes continuous integration, continuous deployment and delivery, and test-driven development.

For this tutorial, we have created a small example application built on top of Falcon, a RESTful API framework for Python. You can find the source code at https://github.com/nimbinatus/circleci-pulumi-sample.

You can use many languages with Pulumi, like Python or Go. For this tutorial, we will deploy it with Pulumi’s TypeScript SDK.

Prerequisites

• A Pulumi account
• The Pulumi SDKs and CLI installed
• A CircleCI account
• A code repo with this code in it

Clone the repo if you prefer, or copy the code. We are explicitly not requiring a specific version control system because Pulumi with CircleCI can use any of them.

Let’s start out with a workflow declaration that deploys our repo to Google Cloud Platform’s (GCP) Container Registry and then to Cloud Run.

Building infrastructure with parameters for declarative workflows

We can parameterize the orb so we can set the version of the Pulumi CLI to download, choose whether the update should skip its preview, and more. With other CI/CD systems, you’d typically need to write one-off Bash scripts to download the client from https://get.pulumi.com, add it to the current $PATH, and so on. Orbs allow for simpler, more composable and declarative CI/CD workflows.

For our example, we can declare our initial workflow cleanly:

version: 2.1
orbs:
  pulumi: pulumi/pulumi@2.1.0
  gcp-cli: circleci/gcp-cli@2.4.0
  gcp-gcr: circleci/gcp-gcr@0.14.1
  node: circleci/node@5.0.0
jobs:
  build:
    docker:
      - image: cimg/node:17.5.0
    steps:
      - checkout
      - setup_remote_docker:
          docker_layer_caching: false
      - pulumi/login:
          access-token: ${PULUMI_ACCESS_TOKEN}
      - node/install-packages:
          app-dir: ~/project/infra-ts/image
      - gcp-cli/install
      - gcp-cli/initialize
      - gcp-gcr/gcr-auth:
          registry-url: us-central1-docker.pkg.dev
      - pulumi/update:
          stack: ${CIRCLE_PROJECT_USERNAME}/docker-image-circleci/dev-personal
          working_directory: ~/project/infra-ts/image
  deploy:
    docker:
      - image: cimg/node:17.5.0
    steps:
      - checkout
      - pulumi/login:
          access-token: ${PULUMI_ACCESS_TOKEN}
      - node/install-packages:
          app-dir: ~/project/infra-ts/cloud-run
      - gcp-cli/install
      - gcp-cli/initialize
      - gcp-gcr/gcr-auth:
          registry-url: us-central1-docker.pkg.dev
      - pulumi/update:
          stack: ${CIRCLE_PROJECT_USERNAME}/circleci-pulumi/dev-personal
          working_directory: ~/project/infra-ts/cloud-run
workflows:
  version: 2.1
  pulumi-deploy:
    jobs:
      - build:
          filters:
            branches:
              only:
                - main
      - deploy:
          requires:
            - build
          filters:
            branches:
              only:
                - main

Note that, throughout this workflow, we’re using standard orb calls with environment variables. No one-off scripts. No mess. Nice, clean, and easy to read, reuse, and share.

Smoke-testing made simple

With Pulumi and CircleCI, you can spin up ephemeral environments to do any testing you want, all as part of your pipeline. We can use parameters to create unique environments, and post the results of the test back up to a pull request or some other spot. For example, we can test pull requests to our GitHub repo using code like this to ensure our endpoint responds as expected, then tear down the testing environment completely once the test has passed. Add this snippet into your .config.yaml file:

# …
jobs:
  preview:
    docker:
      - image: cimg/node:17.5.0
    steps:
      - checkout
      - setup_remote_docker:
          docker_layer_caching: false
      - pulumi/login:
          access-token: ${PULUMI_ACCESS_TOKEN}
      - node/install-packages:
          app-dir: ~/project/infra-ts/cloud-run
      - node/install-packages:
          app-dir: ~/project/infra-ts/image
      - gcp-cli/install
      - gcp-cli/initialize
      - gcp-gcr/gcr-auth:
          registry-url: us-central1-docker.pkg.dev
      - pulumi/stack_init:
          stack: ${CIRCLE_PROJECT_USERNAME}/docker-image-circleci/qa_${CIRCLE_BUILD_NUM}-image
          working_directory: ~/project/infra-ts/image
          copy: ${CIRCLE_PROJECT_USERNAME}/docker-image-circleci/dev-personal
      - pulumi/stack_init:
          stack: ${CIRCLE_PROJECT_USERNAME}/circleci-pulumi/qa_${CIRCLE_BUILD_NUM}-cr
          working_directory: ~/project/infra-ts/cloud-run
          copy: ${CIRCLE_PROJECT_USERNAME}/circleci-pulumi/dev-personal
      - pulumi/preview:
          stack: ${CIRCLE_PROJECT_USERNAME}/docker-image-circleci/qa_${CIRCLE_BUILD_NUM}-image
          working_directory: ~/project/infra-ts/image
      - pulumi/preview:
          stack: ${CIRCLE_PROJECT_USERNAME}/circleci-pulumi/qa_${CIRCLE_BUILD_NUM}-cr
          working_directory: ~/project/infra-ts/cloud-run
      - pulumi/update:
          stack: ${CIRCLE_PROJECT_USERNAME}/docker-image-circleci/qa_${CIRCLE_BUILD_NUM}-image
          working_directory: ~/project/infra-ts/image
          skip-preview: true
      - pulumi/update:
          stack: ${CIRCLE_PROJECT_USERNAME}/circleci-pulumi/qa_${CIRCLE_BUILD_NUM}-cr
          working_directory: ~/project/infra-ts/cloud-run
          skip-preview: true
      - pulumi/stack_output:
          stack: ${CIRCLE_PROJECT_USERNAME}/circleci-pulumi/qa_${CIRCLE_BUILD_NUM}-cr
          property_name: cloudRunUrl
          env_var: TEST_URL
      - run:
          name: Test URL
          command: |
            curl -sSJL $TEST_URL/hello
      - pulumi/destroy:
          stack: ${CIRCLE_PROJECT_USERNAME}/docker-image-circleci/qa_${CIRCLE_BUILD_NUM}-image
          working_directory: ~/project/infra-ts/image
      - pulumi/destroy:
          stack: ${CIRCLE_PROJECT_USERNAME}/circleci-pulumi/qa_${CIRCLE_BUILD_NUM}-cr
          working_directory: ~/project/infra-ts/cloud-run
      - pulumi/stack_rm:
          stack: ${CIRCLE_PROJECT_USERNAME}/docker-image-circleci/qa_${CIRCLE_BUILD_NUM}-image
      - pulumi/stack_rm:
          stack: ${CIRCLE_PROJECT_USERNAME}/circleci-pulumi/qa_${CIRCLE_BUILD_NUM}-cr
# …
workflows:
  version: 2.1
  pulumi-preview:
    jobs:
      - preview:
          filters:
            branches:
              ignore:
                - main
# …

In this workflow, we spin up a new stack for each set of resources, run some tests, and then tear everything down in one pipeline. Very convenient!

Integrating with existing systems

Pulumi’s integration with CircleCI doesn’t end with a basic workflow. The Pulumi command-line client associates CircleCI build metadata with stack updates, so you have links from the Pulumi Service added to any stack updates or previews that happened during a CircleCI workflow.

In addition, you can audit where changes came from since the CircleCI orb shows up as the CI system in use for any changes that came from a CircleCI workflow.

Your Pulumi output shows up in the CircleCI build information on the CircleCI app, as well.

Additionally, if you are using CircleCI with GitHub, hooking Pulumi up with CircleCI will surface the results of any previews or updates from your CI/CD on the source GitHub pull request (PR). It’s always good to know if a pull request is going to lead to changes to your cloud infrastructure!

If you’re on GitHub, head over to the PR to find this view of the preview workflow.

The Details link takes you to the build on CircleCI so you can find all of the information on that build. If you want to have Pulumi add a comment with infrastructure changes on any PR, you can also use the Pulumi GitHub integration to surface any and all changes.

If you’re checking a recent commit, the checks from CircleCI appear when you select the standard checkmark, including all of the workflows you set to run on that branch.

Conclusion

We are always working to make Pulumi the best tool for the “continuous deployment” part of CI/CD workflows, and with CircleCI’s orbs, it’s just that much easier. Try it out by creating a free account and setting up a CircleCI workflow, and let us know what you think!