Control multi-regional infrastructure in a monorepo with CircleCI

How can your organization leverage continuous integration (CI) to manage your infrastructure at scale in a reliable, secure, and repeatable fashion?

If you are maintaining multiple projects, the monorepo strategy can be a useful approach that allows for a unified build and CI process and makes it easier to enforce coding standards and organizational practices. It can also help DevOps and platform teams streamline dependency management to reduce conflicts and compatibility issues.

Many modern infrastructure architectures are complex to deploy, involving many parts. In a previous blog post, we explored how CircleCI can help you manage Kubernetes infrastructure deployments in a secure, resource-friendly manner. In this post, we’ll extend that example to show you how you can automatically provision multi-region infrastructure from a monorepo using a single CircleCI workflow.

Managing Kubernetes infrastructure via a monorepo

Top-performing software engineering organizations prioritize having their applications in a deployable state, and as we discussed in our previous blog, this paradigm is also taking hold in the infrastructure space. Maintaining deployable infrastructure is a competitive differentiator that allows teams to quickly and consistently scale their operations and react to changing demands.

To ensure this process is reliable and repeatable, many organizations leverage the core concepts of CI to automatically build and test their infrastructure on every new commit. This process is known as GitOps.

A common pattern in the world of software engineering is to now split microservices into multiple repositories, and the advantages of this approach are clear. However, scaling a large application across multiple repositories can lead to challenges:

At some point your team could discover that system knowledge is spread across multiple repos maintained by different teams. You may realize that no one knows how to build and deploy the entire system.

In many cases, a monorepo with unified and automated build and deploy pipelines can help mitigate these types of issues.

When building monorepo projects on CircleCI, combining the GitOps approach with best-in-class continuous integration and infrastructure as code (IAC) tools allows you to dynamically maintain, scale, and secure your environments in an automated and resource-efficient way.

If you have decided that a monorepo is the right approach for your team, this post will show you how you can deliver your infrastructure quickly and reliably using an automated CI workflow.

Step 0: Before you begin

To apply the concepts discussed in this tutorial to you own projects, you will need the following resources:

• A CircleCI account
• A GitHub, GitLab, or Bitbucket project that has been set up in CircleCI
• The CircleCI CLI installed on your workstation
• An AWS account
• An EKS cluster
• A monorepo
• Knowledge of Terraform
• Knowledge of Kubernetes

Note: You can find the complete configuration code discussed in this tutorial in this GitHub Gist.

Step 1: Setup config

CircleCI’s dynamic configuration allows you to use jobs and workflows not only to execute work but also to determine what jobs run in response to a given change for more dynamism within your pipelines.

Teams who decide on a monorepo approach for maintaining their infrastructure can leverage dynamic configuration to ensure the delivery is faster, cheaper, and more predictable. Using CircleCI’s path-filtering orb, you can conditionally run a workflow based upon changes made to a specific fileset.

Consider a monorepo structure similar to the following:

To use dynamic configuration, you first need to enable it at the project level. You can find out exactly how to do that in our documentation.

Next, you will need to declare the following to your setup_config.yml file. This will act as your setup configuration file that can dynamically pass parameters to your main configuration file based on the paths changed in your monorepo.

setup: true

More on how dynamic configuration works can be found in our documentation.

As mentioned, we will utilize the path-filtering orb, which allows a pipeline to continue execution based upon the specific paths of updated files. Under the hood, the orb will leverage dynamic config at the beginning of each pipeline execution to determine which paths in the monorepo to build.

orbs: 

  path-filtering: circleci/path-filtering@0.1.7

In our setup config, we will execute one job: path-filtering/filter. As a standard orb practice, we will pass our parameters to the orb job in key-value pairs. The parameters used are:

• base-revision: The revision to compare the current one against for the purpose of determining changed files.

• config-path: The location of the config to continue the pipeline with. Please note that this parameter will be ignored if the user passes the config file per mapping in the mapping parameter.

• mapping: Mapping of path regular expressions to pipeline parameters and values. (We will see these pipeline parameters again in the continue_config.yml file later in the tutorial)

workflows:
  setup-workflow:
    jobs:
      - path-filtering/filter:
          base-revision: main
          config-path: .circleci/continue-config.yml
          mapping: |
            global/.* global true
            namer-eks/.* namer-eks true
            emea-eks/.* emea-eks true
            japac-eks/.* japac-eks true
            namer-platforms/.* namer-platforms true
            emea-platforms/.* emea-platforms true
            japac-platforms/.* japac-platforms true

If the path filtering orb picks up that a path was changed in the global/ subdirectory, for example, the global variable on line 8 will be set to true and passed into the continue_config.yml. We will use this scenario to trigger a specific workflow later in the tutorial.

Step 2: Terraform modules and paths in our monorepo

This is a good time to introduce what each path or subdirectory in our monorepo contains. There are 7 subdirectories in our sample repository, each representing a path on CircleCI:

1. Global

   This module represents common resources needed by all clusters. If this path is triggered, an apply will propagate through all layers of our infrastructure (Global -> EKS -> Platform). Services deployed at the global layer include: Route 53, IAM, and Kuma.

2. namer-eks

3. emea-eks

4. japac-eks

   Our EKS layer and modules will handle the deployment of our EKS clusters in each of the regions above. If this path is triggered, a Terraform apply will trigger the following layers (EKS -> Platform).

   Services deployed at the EKS layer include: An EKS cluster to eu-west-2, us-west-2, or ap-northeast-1 (depending on what region is triggered), Istio service mesh, and Vault.

5. namer-platforms

6. emea-platforms

7. japac-platforms

   Our platform layer and modules will handle the deployment of the required services to our clusters in each of the regions above. If this path is triggered, a Terraform apply will trigger the jobs at the platform level only.

   Services & configuration deployed at the platform layer include: Vault config, Nexus, Nexus Config, Argo Rollouts, and the CircleCI Release Agent.

Below is the flowchart of execution for each path:

And here we can get an insight into how the flowchart above is represented in the repository. Each folder contains the Terraform resources for the module. As such, we can view each folder as a Terraform module. Take a look above to see what each module contains.

As we discussed, the path-filtering/filter job determines which pipeline parameters to update. If a change is made to the global path, then the global variable we see in the mapping stanza of the orb will be set to true. If the orb picks up a change in the namer-eks path, then the namer-eks variable will be set to true, and so on.

Step 3: Continue_config.yaml

The setup config continues the pipeline onto the desired configuration. For this tutorial, the desired configuration is the conntinue_config.yaml file that resides in the same directory. This is a very common pattern and use of the continuation orb.

Orbs

Orbs are a type of package manager for your CircleCI configuration. They provide a lightweight abstraction layer that sits on top of your config and enables you to share common configuration across projects. For this tutorial, we will be using two orbs: the Terraform orb and the AWS CLI orb.

version: 2.1
orbs:
  terraform: circleci/terraform@3.2.1
  aws-cli: circleci/aws-cli@3.1.3

Every CircleCI user gets access to our orb registry, a place where CircleCI experts, partners, and members of the community write and publish orbs.

Parameters

When utilizing CircleCI’s monorepo capabilities, it is typically best to place your CircleCI parameters at the top of the config file. Parameters are reusable pipeline configuration syntax that allow us to dynamically produce pipeline outcomes based on the parameters we pass in.

In the setup config, we introduced parameters as variables passed into the mapping key of the path-filtering job. You will notice that each pipeline parameter default value is false. If a path is changed when a pipeline is triggered, the orb in the setup workflow will cause a true value to be populated in the corresponding pipeline parameter.

parameters:
  run-orb-tests:
    type: boolean
    default: false
  global:
    type: boolean
    default: false
  namer-eks:  
    type: boolean
    default: false
  namer-platforms: 
    type: boolean
    default: false
  emea-eks:  
    type: boolean
    default: false
  emea-platforms: 
    type: boolean
    default: false
  japac-eks:  
    type: boolean
    default: false
  japac-platforms: 
    type: boolean
    default: false

Workflows

A workflow in CircleCI is essentially a job orchestrator. It allows us to introduce a dependency matrix into the execution sequence of our jobs. Workflows support complex job orchestration using a simple set of configuration keys to help you resolve failures sooner.

In Step 2 of this tutorial we discussed the different paths in our monorepo and what workflow a change in each path will subsequently trigger. Each workflow will trigger a different set of jobs, and you can see what workflow each path will trigger in the continue_config.yaml file. As a case in point, we will continue to use the global workflow for this tutorial.

Our global module represents common resources needed by all clusters. If this path is triggered, an apply will propagate through all layers of our infrastructure (Global -> EKS -> Platform).

global-workflow:
    when: << pipeline.parameters.global >>
    jobs:
      - global:
          <<: *tf_job_defaults
      - namer-eks:
          requires:
            - global
          <<: *tf_job_defaults
      - namer-platforms:
          requires:
            - namer-eks
          <<: *tf_job_defaults
          context:
            - CERA-INIT-NAMER
      - emea-eks:
          requires:
            - global
          <<: *tf_job_defaults
      - emea-platforms:
          requires:
            - emea-eks
          <<: *tf_job_defaults
          context:
            - CERA-INIT-EMEA
      - japac-eks:
          requires:
            - global
          <<: *tf_job_defaults
      - japac-platforms:
          requires:
            - japac-eks
          <<: *tf_job_defaults
          context:
            - CERA-INIT-JAPAC

Please refer to the flowchart in section 2 to gain an insight into what each workflow will trigger. Below is a successful execution of the global workflow.

Jobs

Let’s dive into our jobs and see how we are leveraging Terraform to handle the deployment of our infrastructure. We discussed exactly what infrastructure and services will be deployed in each Terraform module/subdirectory. As you can see in the linked configuration file, each job carries out the same steps. We will use the global job to discuss what is being carried out.

The global job will handle the deployment of common resources used by the clusters in all 3 regions: NAMER, EMEA, and JAPAC. If this job is triggered, all layers of the infrastructure will be deployed.

The global job utilizes the Terraform orb and is configured to run on the default executor. This executor is custom built by CircleCI and is highly performant for Terraform deployments. CircleCI provides such executors so teams can begin deploying their Terraform workloads as quickly as possible with best practice in mind.

jobs:
  global:
    executor: terraform/default
    steps:
      - checkout
      - run: mkdir workspace
      - terraform/fmt:
          path: ./global
      - terraform/validate:
          path: ./global
      - terraform/apply:
          path: ./global
      - persist_to_workspace:
          root: workspace
          paths:
            - ./

Next, let’s look at some of the steps in our global job. We can see that for each step, the default Terraform executor is being used.

1. run: mkdir workspace: We are then attaching a workspace that enables CircleCI jobs to share data between one another.

2. terraform/fmt: The Terraform fmt command is used to rewrite Terraform configuration files to a canonical format and style.

3. terraform/validate: This command runs checks that verify whether a configuration is syntactically valid and internally consistent, regardless of any provided variables or existing state

4. terraform/apply: Now are are ready to execute a Terraform apply. We pass in the same resources used in the previous steps, and there you have it. With just this stanza, a Terraform apply will be executed.

5. persist_to_workspace: Finally, we persist the data produced to the workspace, which is passed along the chain of jobs.

tf_job_defaults

You may notice in the Github Gist that there is a piece of code named tf_job_defualts before the job and workflow stanzas are defined. You will also notice that each job defined in a workflow has a pointer to this piece of code.

This code is how we are leveraging OpenID Connect (OIDC) tokens to authenticate to AWS. For more information on why we use OIDC, check out our previous tutorial on managing Kubernetes environments with Gitops and dynamic config. We use pointers so we don’t have to rewrite this code multiple times in our config.

Conclusion

In this tutorial, you learned how to provision your multi-region EKS infrastructure along with other resources in a resource-friendly manner using CircleCI’s dynamic configuration and monorepo capabilities. We have explored various techniques, tips, and strategies to help you master GitOps on CircleCI and increase the agility and scalability of your infrastructure operations. You are now equipped with the knowledge and skills to confidently transform how you handle your infrastructure.

You can find both configuration files discussed in this tutorial on GitHub.

We encourage you to share your feedback and experiences with us on our GitOps support. Your input helps us improve our tutorials and ensure that we continue to provide valuable content that empowers our readers. Feel free to reach out with any questions, suggestions, or success stories. You are an integral part of our learning community.

Start building on CircleCI

Sign up now, connect your VCS repo, and gain the confidence, flexibility, and speed you’ve been looking for.

Start Building for Free