Run private cloud and on-premises jobs with CircleCI runner

CircleCI has released a new feature called CircleCI runner. The runner feature augments and extends the CircleCI platform capabilities and enables developers to diversify their build/workload environments. Diversifying execution environments satisfies some of the specific edge cases mentioned in our CircleCI runner announcement. For example, some of our larger customers in highly regulated industries, like finance and healthcare, must meet compliance requirements that prevent them from running some workloads in the cloud. Others working on embedded systems or IoT need to build on hardware that simply does not exist in the cloud. We built the runner so that even customers with the strictest security and compliance requirements can meet 100% of their software delivery needs with CircleCI. In this post I will introduce the CircleCI runner and demonstrate how to use runners in your pipelines.

Prerequisites

Before you can get started with the runner, you need to complete a number of tasks:

• Create a DigitalOcean Account
  • Create and upload a Public SSH Key to DigitalOcean account
  • Generate a DigitalOcean API Token
  • Optional: Install the DigitalOcean doctl command line tool locally
• Install Terraform CLI locally
  • Create a Terraform Cloud Account
  • Create a new Terraform Cloud organization
  • Create 2 new Terraform Cloud workspaces named dorunner and choose the "No VCS connection" option
  • Enable local execution mode in the dorunner workspace
  • Create a new Terraform API token
• Git clone this circleci-runners example repo from GitHub
• Sign-up for a CircleCI account
• Complete all of the steps in the runner Authentication section
  • Install the CircleCI CLI tool locally
  • Create a [Personal API Token][18] and authenticate/configure the cli tool
  • Create a namespace for your organization’s runner resources.
  • Create a resource class for your runner for your namespace.
  • Create a token for authenticating the above resource-class.

Runner node platforms

The current offering of CircleCI runner installs and operates on theses platforms and operating systems:

Installation target	Target Value
Linux x86_64	platform=linux/amd64
Linux ARM64	platform=linux/arm64
macOS x86_64	platform=darwin/amd64

In this tutorial we will be using Ubuntu 20.04 and the platform=linux/amd64 platform.

Provisioning CircleCI runner configurations

Before deploying runner nodes and processing jobs, you will need to create namespaces and resource classes, and generate runner tokens. These actions were mentioned in the Prerequisites section of this tutorial, but I’ll cover them in a bit more detail here.

Create CircleCI namespace

CircleCI requires you to create a namespace on the platform that defines your unique identity within the CircleCI platform. This namespace is also used for other features such as CircleCI Orbs registry. If you’ve already created a namespace you can move forward to the next section.

Use the CircleCI CLI tool to create a new namespace:

circleci namespace create <your namespace> <vcs-type> <org-name>

Specify your namespace name/value and your vcs/version control type (github etc..). Then specify your organization name (generally your GitHub user/organization name).

Create a new resource class

After creating your namespace, you need to create a new resource class for your runners. Resource classes let you group similar runner resources, and can be used to classify your runners. For example, if you expect to use macOS and Linux runners, you should create two resource classes, one for macOS resources and one for Linux resources. You can also classify resource classes in relation to the compute nodes hardware resources such as CPU, RAM, disk, and network capabilities. Run this command to create a new runner resource class:

circleci runner resource-class create <my-namespace/resource-class> <description>

The section <my-namespace/resource-class> requires you to specify your CircleCI namespace, and then create a resource class name. In this example: punkdata/do-ub-linux-cpu4-16gbram, the namespace is punkdata and do-ub-linux-cpu4-16gbram is the resource class name. do-ub-linux-cpu4-16gbram is a descriptive name that represents an Ubuntu Linux platform using a DigitalOcean compute node with four CPUs and 16 GB of RAM. If you need to, you can give more detail about the resource class by populating the <description> parameter.

Create a runner token

The last task to complete using the CLI tool is generating the runner token used by the runner nodes to authenticate against the CircleCI platform.

Note: The token cannot be retrieved again, so be sure to store it safely after you create it.

Generate the runner token by running this command:

circleci runner token create <my-namespace/resource-class> <nickname>

As before, specify your <my-namespace/resource-class> parameter values, and then specify a descriptive and memorable <nickname> value for your token.

CircleCI runner manual installation

CircleCI runners use a manual installation process that details how to install and configure the required software on the corresponding compute nodes. I encourage you to familiarize yourself with this manual process. For this tutorial, though, we will use Terraform to provision CircleCI runners on the DigitalOcean platform.

CircleCI runners provision with automation

Next I will cover the automated runner provisioning process that I built using Terraform. The code available from this repo is what we will be using to provision CircleCI runners. I hope that you have completed the steps in the Prerequisites section. You need to have them finished by now so that you can complete this tutorial .

The relevant code in this project repo structure is laid out in these directories:

runner-scripts/
  |_runner-agent-install
  |_runner-provisioner
terraform/
  |_do/
      |_main.tf
      |_variables.tf

In the next section I will discuss the contents of the runner-agent_install and runner-provisioner files. These files automate some of the manual runner installation processes.

Runner scripts

The runner-scripts/ directory contains script files that automate the installation and provisioning of runner nodes. These scripts are made up of code and commands that encapsulate the various scripts and commands defined in the manual runner installation process. These scripts can be used, without changes, to install and provision runner nodes. In this tutorial we will leverage them in our Terraform code. Here is a quick breakdown of these files.

The runner-agent-install script has a “platform” parameter and must be assigned one of these values:

• platform=linux/amd64
• platform=linux/arm64
• platform=darwin/amd64

This script must be executed first to install the appropriate runner agent.

The runner-provisioner script provisions and configures the runner agent and underlying operating system services. This script has parameters that must be assigned:

• PLATFORM needs to be one of the supported platform values listed above
• RUNNER_NAME identifies the runner node; I recommend using the node’s hostname
• RUNNER_TOKEN is the runner token you generated a few steps back in this tutorial

This script automates runner configuration and OS level actions, like user provisioning and runner-related service implementations. Running these scripts on a compute node will produce a fully functional CircleCI runner node that can be used as executor in your CircleCI configurations.

Next I will discuss the Terraform code that leverages these scripts to provision runner nodes using infrastructure as code.

Terraform runner provisioner code

Let me take a moment to walk through the Terraform code that provisions runner nodes. Take a look at the files in the terraform/do/ directory, starting with the variables.tf file.

The variables.tf file holds all the variables that the Terraform project uses. The variable definitions specified in this file enable passing values using the Terraform CLI. Here is an example of the variables.tf file:

variable "do_token" {
  type        = string
  description = "Your DigitalOcean API token. See https://cloud.digitalocean.com/account/api/tokens to generate a token."
}

variable ssh_key_file {
  type        = string
  description = "The private ssh certificate in the user's .ssh/ dir -> $HOME/.ssh/id_rsa. Terrform uses this to access & run commands on node."
}

variable do_ssh_key {
  type        = string
  default     = "ariv3ra-ssh"
  description = "Specifies a Public SSH cert that exists in the DO Account Security section. See https://www.digitalocean.com/docs/droplets/how-to/add-ssh-keys/to-account/"
}

variable file_agent_install {
  type        = string
  default     = "../../runner-scripts/runner-agent-install"
  description = "Specifies the runner-agent-install script file path"
}

variable file_provisioner {
  type        = string
  default     = "../../runner-scripts/runner-provisioner"
  description = "Specifies the runner-provisioner script file path"
}

variable runner_platform {
  type        = string
  default     = "linux/amd64"
  description = "Defines the runner architecture platform choose one: [linux/amd64, linux/arm64, darwin/amd64]"
}

variable runner_name {
  type        = string
  description = "The host name of the CircleCI Runner which is also the Droplet name."
}

variable runner_token {
  type        = string
  description = "The CircleCI Runner token from the CLI"
}

variable "region" {
  type        = string
  description = "The region where the assets should be assigned to"
  default     = "nyc3"
}

variable "node_size" {
  type        = string
  description = "defines the size of the compute node"
  default     = "s-4vcpu-8gb"
}

variable "image_name" {
  type        = string
  description = "defines the DigitalOcean image name to install to compute node"
  default     = "ubuntu-20-04-x64"
}

variable droplet_count {
  type        = number
  default     = 2
  description = "The number of Droplet nodes you want to create"
}

The variables listed in this example have description fields that tell you what each variable specifies. Using these variables you can assign values to things like runner node counts, compute node sizes, runner platform, and any other relevant data about runner or cloud providers. In this case, the Terraform code pertains to provisioning runner nodes in the form of DigitalOcean Droplets, which are essentially compute nodes.

Next, have a look at the main.tf file, which contains the core Terraform functionality for this project. Here is an example of the main.tf file:

terraform {

  required_version = ">= 0.13"

  required_providers {
    digitalocean = {
      source = "digitalocean/digitalocean"
    }
    local = {
      source = "hashicorp/local"
    }
  }
  # This backend uses the terraform cloud for state.
  backend "remote" {
    organization = "datapunks"
    workspaces {
      name = "dorunner"
    }
  }
}

provider "digitalocean" {
  token = var.do_token
}

data "digitalocean_ssh_key" "terraform" {
  name = var.do_ssh_key
}

resource "digitalocean_droplet" "dorunner" {
  count              = var.droplet_count
  image              = var.image_name
  name               = "${var.runner_name}-${count.index}"
  region             = var.region
  size               = var.node_size
  private_networking = true
  ssh_keys = [
    data.digitalocean_ssh_key.terraform.id
  ]
  connection {
    host        = self.ipv4_address
    user        = "root"
    type        = "ssh"
    private_key = file(var.ssh_key_file)
    timeout     = "3m"
  }

  #Upload runner agent install script
  provisioner "file" {
    source      = var.file_agent_install
    destination = "/tmp/runner-agent-install"
  }

  #Upload runner provisioner script
  provisioner "file" {
    source      = var.file_provisioner
    destination = "/tmp/runner-provisioner"
  }

  provisioner "remote-exec" {
    inline = [
      "export PATH=$PATH:/usr/bin",
      "sudo apt update",
      "sudo apt -y upgrade",
      "curl -sL https://deb.nodesource.com/setup_14.x | sudo bash -",
      "sudo apt install -y nodejs",
      "cd /tmp",
      "chmod +x /tmp/runner-agent-install",
      "chmod +x /tmp/runner-provisioner",
      "/tmp/runner-agent-install ${var.runner_platform}",
      "/tmp/runner-provisioner ${var.runner_platform} ${var.runner_name} ${var.runner_token}",
    ]
  }
}

output "runner_hosts and ip_addresses" {
  value = {
    for instance in digitalocean_droplet.dorunner :
    instance.name => instance.ipv4_address
  }
}

This file is composed of multiple elements. I will break them out and provide a description following each code snippet.

terraform {

  required_version = ">= 0.13"

  required_providers {
    digitalocean = {
      source = "digitalocean/digitalocean"
    }
    local = {
      source = "hashicorp/local"
    }
  }
  # This backend uses the terraform cloud for state.
  backend "remote" {
    organization = "datapunks"
    workspaces {
      name = "dorunner"
    }
  }
}

This snippet specifies the Terraform provider (DigitalOcean) that we are targeting. It also specifies where the project state will be maintained. In this case, it is in the dorunner Terraform Cloud workspace that you created in the Prerequisites section.

provider "digitalocean" {
  token = var.do_token
}

data "digitalocean_ssh_key" "terraform" {
  name = var.do_ssh_key
}

This snippet specifies the DigitalOcean API token and the SSH Key configured in the DigitalOcean platform. These values are specified using the variables declared in the variables.tf file.

resource "digitalocean_droplet" "dorunner" {
  count              = var.droplet_count
  image              = var.image_name
  name               = "${var.runner_name}-${count.index}"
  region             = var.region
  size               = var.node_size
  private_networking = true
  ssh_keys = [
    data.digitalocean_ssh_key.terraform.id
  ]
  connection {
    host        = self.ipv4_address
    user        = "root"
    type        = "ssh"
    private_key = file(var.ssh_key_file)
    timeout     = "3m"
  }

This code snippet represents the automation that actually creates and provisions runner nodes. The major elements you need to know about:

• name is an interpolation of existing variables which generates unique runner node host names
• ssh_keys uses the data.digitalocean_ssh_key block to retrieve the public ssh key from DO
• Terraform requires access to execute commands on newly created nodes. private_key specifies the private ssh key that corresponds with the public ssh key specified in the var.do_ssh_key variable

  #Upload runner agent install script
  provisioner "file" {
    source      = var.file_agent_install
    destination = "/tmp/runner-agent-install"
  }

  #Upload runner provisioner script
  provisioner "file" {
    source      = var.file_provisioner
    destination = "/tmp/runner-provisioner"
  }

This snippet uploads runner_agent_install and runner_provisioner to the newly created compute node. These files will install the CircleCI runner agent and provision the node as a CircleCI runner node.

  provisioner "remote-exec" {
    inline = [
      "export PATH=$PATH:/usr/bin",
      "sudo apt update",
      "sudo apt -y upgrade",
      "curl -sL https://deb.nodesource.com/setup_14.x | sudo bash -",
      "sudo apt install -y nodejs",
      "cd /tmp",
      "chmod +x /tmp/runner-agent-install",
      "chmod +x /tmp/runner-provisioner",
      "/tmp/runner-agent-install ${var.runner_platform}",
      "/tmp/runner-provisioner ${var.runner_platform} ${var.runner_name} ${var.runner_token}",
    ]
  }
}

This snippet remotely executes commands on the compute nodes. The inline parameter is a list of commands to execute on the compute node. These commands make up the provisioning process for CircleCI runner nodes. This block has important commands that I would like to point out:

• curl -sL https://deb.nodesource.com/setup_14.x | sudo bash - downloads Node.js version 14
• /tmp/runner-agent-install ${var.runner_platform} executes the runner install script using the var.runner_platform value
• /tmp/runner-provisioner ${var.runner_platform} ${var.runner_name} ${var.runner_token} executes the runner provisioning script using multiple variables such as platform, runner name and token

output "runner_hosts_and_ip_addresses" {
  value = {
    for instance in digitalocean_droplet.dorunner :
    instance.name => instance.ipv4_address
  }
}

Finally, this snippet is the Terraform output that iterates over all the nodes and prints the host names and corresponding IP Addresses. When the Terraform execution is complete, you will have fully functional CircleCI runner nodes ready to begin processing your runner pipeline workloads.

Create CircleCI runner nodes

I’ll be using the example code repo to demonstrate creating runner nodes in DigitalOcean.

In a terminal type:

cd terraform/do

terraform init

The previous command will initialize the Terraform project. The next command will execute the Terraform creation process:

terraform apply \
  -var "do_token=${DIGITAL_OCEAN_TOKEN}" \
  -var "runner_token=${RUNNER_TOKEN}" \
  -var "runner_name=dorunner" \
  -var "ssh_key_file=${HOME}/.ssh/id_rsa" \
  -auto-approve

This Terraform command specifies values for Terraform variables. I recommend assigning sensitive data to local system environment variables to protect against exposure and to maintain consistent behavior when working in terminals. If the Terraform variables have default values assigned, those will be used and are not required. You can override the default variable values by simply specifying the variable you want to override and supplying the new value to be used.

After running this command you should see results similar to this:

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

Outputs:

runner_hosts_and_ip_addresses = {
  "dorunner-0" = "68.183.55.75"
  "dorunner-1" = "68.183.61.144"
}

As you can see here, two individual DigitalOcean Droplet nodes have been created and provisioned as CircleCI runner nodes. These nodes are ready to begin processing pipeline workloads. The results show the nodes’ “host names” and corresponding IP Addresses, which you can access using SSH if you need to.

Modifying runner nodes

Now that you have created runner nodes with Terraform, you can also use Terraform to manage the nodes. For example, I initially created two runner nodes to process pipeline workloads. If it turns out that two nodes are not enough to process all the workloads, you can add more. If you need to, you can add three additional runner nodes to assist in processing workloads. In this case, you can adjust the var.droplet_count value to be the amount of nodes you need. Why not double the amount to 4 active runners so we have ample nodes to process workloads? Here is an example command that you could use to add 2 new runner nodes:

terraform apply \
  -var "do_token=${DIGITAL_OCEAN_TOKEN}" \
  -var "runner_token=${RUNNER_TOKEN}" \
  -var "runner_name=dorunner" \
  -var "ssh_key_file=${HOME}/.ssh/id_rsa" \
  -var "droplet_count=4" \
  -auto-approve

I used this command to change the existing runner node count from 2 to 4 by assigning the new amount to the Terraform variable -var "droplet_count=4" in the apply command. After running the command you should see results like this:

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

Outputs:

runner_hosts_and_ip_addresses = {
  "dorunner-0" = "68.183.55.75"
  "dorunner-1" = "68.183.61.144"
  "dorunner-2" = "64.225.21.96"
  "dorunner-3" = "64.225.21.95"
}

In these results you can see that two new runner nodes have been added. You now have a total of four runner nodes ready to process workloads. You can use this approach reduce the total amount of nodes you require. In any case, the droplet-count variable is the control point for node totals.

CircleCI runner example config.yml

So now that we have some CircleCI runner nodes provisioned and at the ready, we can start using them to process workloads in our CI/CD pipelines. I have created an example config.yml that demonstrates how to use CircelCI runner nodes in pipeline jobs:

version: 2.1

jobs:

  runner_build_test:
    machine: true
    resource_class: datapunks/dorunner
    steps:
      - checkout
      - run:
          name: Install npm dependencies
          command: |
            npm install
      - run:
          name: Run Unit Tests
          command: |
            ./node_modules/mocha/bin/mocha test/ --reporter mochawesome --reporter-options reportDir=test-results,reportFilename=test-results
      - store_artifacts:
          path: test-results

workflows:
  build_test_deploy:
    jobs:
      - runner_build_test

This example shows a simple job that executes on a runner node. The runner_build_test: job shows how to define runner executors using the machine: and resource_class: keys. The resource_class: key is assigned using the namespace/resource_class_name combo. This combination must be used when specifying the values to the resource_class: key. The example is from a Node.js project and requires Node.js to install on the runner nodes, which occurred in the Terraform provisioning process.

Now that I demonstrated how to implement and use runners within your config.yml files, you should be well informed about how to easily get CircleCI runner nodes up and running using Terraform on the platforms and cloud providers of your choice. As a bonus, I will use the next section to describe some tips you can use to help get runners ready to go.

CircleCI runner tips

CircleCI runners are still very new, so I would like to share some tips that you may find helpful when using runners:

• Install software on runner nodes. Runner must have all the software that your builds are dependent on installed locally: Node.js, git, gcc, python, Docker client, and whatever else is needed.
• Understand runner security policies. If you are running nodes in cloud providers, be sure you fully understand the intricacies of security within the respective providers. Understanding things like security groups, roles-based access, and firewall configurations are critical in operating runner nodes.
• Patch your runners. Runners are essentially compute nodes that augment your current CircleCI pipelines. Management of these nodes falls solely on your organization. This includes updating appropriate software/dependencies and operating system security and vulnerability patches. Maintaining up-to-date software and secure runner nodes is critical for protecting your workloads.
• Understand runner limitations. Get familiar with the current runner limitations here, so you understand what runner can and cannot do.

Conclusion

The CircleCI platform handles all CI/CD processing, but CircleCI runners are a great way to augment your CI/CD pipelines. Runners can be especially helpful if you need to to perform specialized processing, or if you have custom or regulated requirements. Use runners only to fulfill specific use cases, like a requirement to run jobs on on-premises. Runners can help when there is limited-access infrastructure due to stricter isolation, or if you use a unique compute architecture that CircleCI does not offer as a resource class.

In this post, I gave a brief introduction and demonstrated how to automate creating and provisioning CircleCI runners using script and Terraform. I also demonstrated how to define runner jobs in pipelines. I would love to hear your feedback, thoughts and opinions so please join the discussion by tweeting to me @punkdata.

Thanks for reading!