Deploy autoscaling self-hosted runners using AWS CDK

This tutorial covers:

1. Defining the AWS CDK application and the AWS Lambda handler
2. Creating a self-hosted runner resource class
3. Deploying the application and then automating the deployments

You can use CircleCI’s cloud resources to run your CI/CD jobs, but there may be times you want to run them on your infrastructure. If your team imposes privileged access and control requirements, a self-hosted infrastructure might be best for running your jobs. CircleCI’s self-hosted runners lets you do exactly that. It is easy to get started and start using self-hosted runners.

If you find that your team has resource requirements that fluctuate daily, you could consider implementing an autoscaling solution to spin up the resources based on the demand.

In this tutorial, you will learn how to set up autoscaling using AWS CDK, an Infrastructure as Code (IaC) tool developed by AWS. If you are interested in implementing autoscaling using the AWS GUI console, there is a companion tutorial.

Prerequisites

Refer to this list to set up everything you need for this tutorial.

• Install NodeJS and NPM
• Install AWS CLI and configure AWS credentials
• Install AWS CDK CLI.
• Create an AWS account and a CircleCI account.
• Create a CircleCI personal access token
• Install self-hosted runner using CircleCI web app.

Create a new AWS CDK project

Create a new directory for your CDK project and navigate into it.

mkdir circleci-self-hosted-runner-autoscaling
cd circleci-self-hosted-runner-autoscaling

Use the CDK CLI to run the cdk init command, which creates a new CDK project using Typescript. The app parameter specifies the template you will use for initializing the project.

cdk init app --language typescript

This command creates a new CDK project with a few files.

Make sure that aws-cdk-lib >= 2.32.0 is defined in the package.json. Manually update the CDK library version if the existing version is lower.

Add Lambda function for autoscaling

In this section, you will define an AWS Lambda function to set the desired number of running instances based on the current demand.

Create a lambda/auto-scaling-lambda directory at the root of the CDK project. Inside the lambda/auto-scaling-lambda directory, add a package.json file for defining the dependencies.

In the package.json file, define the project’s name and add the node-fetch dependency that your handler will use.

{
  "name": "auto-scaling-lambda",
  "version": "0.1.0",
  "dependencies": {
    "node-fetch": "^2.6.7"
  }
}

Once you add the dependencies, run the npm install command in the lambda/auto-scaling-lambda directory to install the packages.

Next, add an index.js file in the lambda/auto-scaling-lambda directory to define the Lambda handler. The function will call the CircleCI runner tasks API to fetch the number of pending jobs. Based on the response, it will update the instance count in the AWS autoscaling group.

Add this code to the index.js file:

const AWS = require("aws-sdk");
const fetch = require("node-fetch");
AWS.config.update({ region: "us-west-2" });
const { env } = require("process");

const SECRET_NAME = env.SECRET_NAME;
const SECRET_REGION = env.SECRET_REGION;
const AUTO_SCALING_MAX = env.AUTO_SCALING_MAX;
const AUTO_SCALING_GROUP_NAME = env.AUTO_SCALING_GROUP_NAME;
const AUTO_SCALING_GROUP_REGION = env.AUTO_SCALING_GROUP_REGION;

exports.handler = async (event, context) => {
  return await getTasks().then(async (data) => {
    let numInstances = 0;
    if (data["unclaimed_task_count"] < AUTO_SCALING_MAX) {
      numInstances = data["unclaimed_task_count"];
    } else {
      numInstances = AUTO_SCALING_MAX;
    }

    await updateNumInstances(numInstances);
    return numInstances;
  });
};

async function updateNumInstances(numInstances) {
  const autoScaling = new AWS.AutoScaling({ region: AUTO_SCALING_GROUP_REGION });
  const params = {
    AutoScalingGroupName: AUTO_SCALING_GROUP_NAME,
    MinSize: 0,
    MaxSize: AUTO_SCALING_MAX,
    DesiredCapacity: numInstances,
  };
  await autoScaling.updateAutoScalingGroup(params).promise();
}

async function getTasks() {
  const secret = await getSecret();
  const url = `https://runner.circleci.com/api/v2/tasks?resource-class=${secret["resource_class"]}`;
  const headers = {
    "Circle-Token": secret["circle_token"],
  };

  const response = await fetch(url, {
    headers: headers,
  });
  const data = await response.json();
  return data;
}

async function getSecret() {
  const params = {
    SecretId: SECRET_NAME,
  };
  const data = await new AWS.SecretsManager({ region: SECRET_REGION })
    .getSecretValue(params)
    .promise();
  if ("SecretString" in data) {
    let secret = JSON.parse(data.SecretString);
    return secret;
  } else {
    let buff = new Buffer(data.SecretBinary, "base64");
    let decodedBinarySecret = buff.toString("ascii");
    return JSON.parse(decodedBinarySecret);
  }
}

Please note:

• The Lambda function uses AWS Secrets Manager to retrieve the CircleCI token required for authenticating the tasks API. It receives the name of the secret as an environment variable.
• The function also receives the AWS EC2 auto scaling group name as an environment variable. It uses NodeJS’s AWS SDK to call autoscaling service API to update the instance count.

Define the AWS EC2 startup script

You need to configure and run the CircleCI self-hosted runner service on your AWS EC2 instances as soon as it boots up. Define a shell script that installs the required packages, creates the CircleCI runner service, and starts the service to make it discoverable. Create a file at scripts/install_runner.sh and add this code snippet:

#!/bin/bash

#-------------------------------------------------------------------------------
# CircleCI Runner installation script
# Based on the documentation at https://circleci.com/docs/2.0/runner-installation/
#-------------------------------------------------------------------------------

# Prerequisites:
# Complete these:
# https://circleci.com/docs/2.0/runner-installation/#authentication
# This script must be run as root
# This script was tested on Ubuntu 22.04

platform="linux/amd64"                                  # Runner platform: linux/amd64 || linux/arm64 || platform=darwin/amd64
prefix="/opt/circleci"                                  # Runner install directory

CONFIG_PATH="/opt/circleci/launch-agent-config.yaml"    # Determines where Runner config will be stored
SERVICE_PATH="/opt/circleci/circleci.service"           # Determines where the Runner service definition will be stored
TIMESTAMP=$(date +"%g%m%d-%H%M%S-%3N")                  # Used to avoid Runner naming collisions

AUTH_TOKEN="<SELF_HOSTED_RUNNER_AUTH_TOKEN>"  # Auth token for CircleCI
RUNNER_NAME="<SELF_HOSTED_RUNNER_NAME>"           # A runner name - this is not the same as the Resource class - keep it short, and only with letters/numbers/dashes/underscores
UNIQUE_RUNNER_NAME="$RUNNER_NAME-$TIMESTAMP"            # Runners must have a unique name, so we'll append a timestamp
USERNAME="circleci"                                     # The user which the runner will execute as

#-------------------------------------------------------------------------------
# Update; install dependencies
#-------------------------------------------------------------------------------

apt update
apt install coreutils curl tar gzip -y

#-------------------------------------------------------------------------------
# Download, install, and verify the binary
#-------------------------------------------------------------------------------

sudo mkdir -p "$prefix/workdir"
base_url="https://circleci-binary-releases.s3.amazonaws.com/circleci-launch-agent"
echo "Determining latest version of CircleCI Launch Agent"
agent_version=$(curl "$base_url/release.txt")
echo "Using CircleCI Launch Agent version $agent_version"
echo "Downloading and verifying CircleCI Launch Agent Binary"
curl -sSL "$base_url/$agent_version/checksums.txt" -o checksums.txt
file="$(grep -F "$platform" checksums.txt | cut -d ' ' -f 2 | sed 's/^.//')"
sudo mkdir -p "$platform"
echo "Downloading CircleCI Launch Agent: $file"
curl --compressed -L "$base_url/$agent_version/$file" -o "$file"
echo "Verifying CircleCI Launch Agent download"
grep "$file" checksums.txt | sha256sum --check && chmod +x "$file"; sudo cp "$file" "$prefix/circleci-launch-agent" || echo "Invalid checksum for CircleCI Launch Agent, please try download again"

#-------------------------------------------------------------------------------
# Install the CircleCI runner configuration
# CircleCI Runner will be executing as the configured $USERNAME
# Note the short idle timeout - this script is designed for auto-scaling scenarios - if a runner is unclaimed, it will quit and the system will shut down as defined in the below service definition
#-------------------------------------------------------------------------------

sudo bash -c 'cat > /opt/circleci/launch-agent-config.yaml' << EOF
api:
  auth_token: $AUTH_TOKEN
runner:
  name: $UNIQUE_RUNNER_NAME
  command_prefix: ["sudo", "-niHu", "$USERNAME", "--"]
  working_directory: /opt/circleci/workdir/%s
  cleanup_working_directory: true
  idle_timeout: 5m
  max_run_time: 5h
  mode: single-task
EOF

# Set correct config file permissions and ownership
chown root: /opt/circleci/launch-agent-config.yaml
chmod 600 /opt/circleci/launch-agent-config.yaml

#-------------------------------------------------------------------------------
# Create the circleci user & give permissions to working directory
# This user should NOT already exist
#-------------------------------------------------------------------------------

adduser --disabled-password --gecos GECOS "$USERNAME"
chown -R "$USERNAME" "$prefix/workdir"

#-------------------------------------------------------------------------------
# Create the service
# The service will shut down the instance when it exits - that is, the runner has completed with a success or error
#-------------------------------------------------------------------------------

sudo bash -c 'cat > /opt/circleci/circleci.service' << EOF
[Unit]
Description=CircleCI Runner
After=network.target
[Service]
ExecStart=$prefix/circleci-launch-agent --config $CONFIG_PATH
ExecStopPost=shutdown now -h
Restart=no
User=root
NotifyAccess=exec
TimeoutStopSec=18300
[Install]
WantedBy = multi-user.target
EOF

#-------------------------------------------------------------------------------
# Configure your runner environment
# This script must be able to run unattended - without user input
#-------------------------------------------------------------------------------
sudo apt install -y nodejs npm

#-------------------------------------------------------------------------------
# Enable CircleCI Runner service and start it
# This MUST be done last, as it will immediately advertise to the CircleCI server that the runner is ready to use
#-------------------------------------------------------------------------------
sudo systemctl enable $prefix/circleci.service
sudo systemctl start circleci.service

Note that the script contains <SELF_HOSTED_RUNNER_AUTH_TOKEN> and <SELF_HOSTED_RUNNER_NAME> as dummy values, which will be replaced with values from the environment variables. In the next section, the shell script will be used by the AWS EC2 autoscaling group in the CDK stack.

Define CDK Constructs for the application

In this section, you will define all the CDK constructs for your application. AWS CDK Constructs are cloud components encapsulating the configuration detail and gluing logic for using one or multiple AWS services. CDK provides a library of constructs for the most commonly used AWS services.

When you generate the CDK project using the app template, the lib/circleci-self-hosted-runner-autoscaling-stack.ts file is created for you containing the CircleciSelfHostedRunnerAutoscalingStack class. You will be defining the CDK constructs in this file.

//  The snippet shows the original contents for reference. You do not need to replace the file contents.
import { Stack, StackProps } from "aws-cdk-lib";
import { Construct } from "constructs";

export class CircleciSelfHostedRunnerAutoscalingStack extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);

    // we will add all the constructs here
  }
}

First, extend StackProps to CircleciSelfHostedRunnerAutoscalingStackProps and define a few additional properties. You are extending the StackProps so that the stack can receive a few custom parameters from the CDK app.

import { Stack, StackProps } from "aws-cdk-lib";
import { Construct } from "constructs";

// extend StackProps
export interface CircleciSelfHostedRunnerAutoscalingStackProps extends StackProps {
  maxInstances: string;
  keypairName: string;
  runnerName: string;
}

export class CircleciSelfHostedRunnerAutoscalingStack extends Stack {
  // use CircleciSelfHostedRunnerAutoscalingStackProps instead of StackProps
  constructor(scope: Construct, id: string, props?: CircleciSelfHostedRunnerAutoscalingStackProps) {
    super(scope, id, props);
  }
}

Next, list all the constructs you need to define in the stack for autoscaling:

• A VPC, a security group, and an AMI for your AWS EC2 instances.
• A autoscaling group for AWS EC2 which executes a custom user script on boot up.
• Secret using AWS secret manager for storing the CircleCI token and resource class.
• An AWS Lambda function which runs every minute to update the desired instance count.

Define constructs for AWS EC2

In this section you will define AWS CDK constructs for things needed when creating an AWS EC2 autoscaling group. You need to define a VPC, a security group, and an AMI for your EC2 instance.

To define a VPC, add a CDK construct inside the constructor defined in the lib/circleci-self-hosted-runner-autoscaling-stack.ts file.

import { Stack,
    StackProps,
    //update the existing import to add aws_ec2
    aws_ec2 as ec2,
 } from 'aws-cdk-lib';

constructor(scope: Construct, id: string, props?: CircleciSelfHostedRunnerAutoscalingStackProps) {
    super(scope, id, props);

    // we will add all the constructs here
    // provide a unique name for your S3 bucket
    const circleCIVpc = new ec2.Vpc(this, "CircleCISelfHostedRunnerVPC", {
        maxAzs: 1,
        subnetConfiguration: [{
        name: 'public-subnet-1',
        subnetType: ec2.SubnetType.PUBLIC,
        cidrMask: 24,
        }]
    });
}

To define a security group, add this code snippet inside the constructor defined in the lib/circleci-self-hosted-runner-autoscaling-stack.ts file.

import { Stack,
    StackProps,
    //update the existing import to add aws_ec2
    aws_ec2 as ec2,
 } from 'aws-cdk-lib';

constructor(scope: Construct, id: string, props?: CircleciSelfHostedRunnerAutoscalingStackProps) {
    super(scope, id, props);

    // add the security group below the existing constructs
    const circleCISecurityGroup = new ec2.SecurityGroup(this, 'CircleCISelfHostedRunnerSecurityGroup', {
        vpc: circleCIVpc,
    });

    circleCISecurityGroup.addIngressRule(
        ec2.Peer.anyIpv4(),
        ec2.Port.tcp(22),
        'allow SSH access from anywhere',
    );
}

To define a AMI for the EC2 instances, add this code snippet inside the constructor defined in the lib/circleci-self-hosted-runner-autoscaling-stack.ts file. A file is available AWS SAM parameter for getting the AMI ID of the Ubuntu instance for the selected AWS region.

import { Stack,
    StackProps,
    //update the existing import to add aws_ec2
    aws_ec2 as ec2,
 } from 'aws-cdk-lib';

constructor(scope: Construct, id: string, props?: CircleciSelfHostedRunnerAutoscalingStackProps) {
    super(scope, id, props);

    // add the AMI below the existing constructs
    const amiSamParameterName = '/aws/service/canonical/ubuntu/server/focal/stable/current/amd64/hvm/ebs-gp2/ami-id'

    const ami = ec2.MachineImage.fromSsmParameter(
      amiSamParameterName, {
      os: ec2.OperatingSystemType.LINUX
    });
}

Define an autoscaling group

Next, define an AWS EC2 autoscaling group for managing your instances. Add this code snippet inside the constructor defined in the lib/circleci-self-hosted-runner-autoscaling-stack.ts file.

import {
    StackProps,
    aws_ec2 as ec2,
    //update the existing import to add aws_autoscaling
    aws_autoscaling as autoscaling,
} from 'aws-cdk-lib';

constructor(scope: Construct, id: string, props ?: CircleciSelfHostedRunnerAutoscalingStackProps) {

    // add the following code snippet below the existing constructs
    const instanceTypeName = "t3.micro"
    const instanceType = new ec2.InstanceType(instanceTypeName);

    const circleCiAutoScalingGroup = new autoscaling.AutoScalingGroup(this, 'CircleCiSelfHostedRunnerASG', {
        vpc: circleCIVpc,
        instanceType: instanceType,
        machineImage: ami,
        securityGroup: circleCISecurityGroup,
        keyName: props!!.keypairName,
        vpcSubnets: {
            subnetType: ec2.SubnetType.PUBLIC
        },
        minCapacity: 0,
        maxCapacity: Number(props!!.maxInstances),
    });
}

If you want to SSH into the EC2 instances, you need to manually create an EC2 key pair using the AWS console. Otherwise, you can omit the keyName property while defining the autoscaling construct.

The autoscaling CDK construct allows you to attach a user data script that will execute on instance startup. Attach the user data script to the autoscaling group by adding this code snippet in the stack file:

import { readFileSync } from 'fs';
const { env } = require("process");

constructor(scope: Construct, id: string, props ?: CircleciSelfHostedRunnerAutoscalingStackProps) {

    // add the following code snippet below the existing constructs
    let userDataScript = readFileSync('./scripts/install_runner.sh', 'utf8');

    userDataScript = userDataScript.replace('<SELF_HOSTED_RUNNER_AUTH_TOKEN>', env.SELF_HOSTED_RUNNER_AUTH_TOKEN);
    userDataScript = userDataScript.replace('<SELF_HOSTED_RUNNER_NAME>', props!!.runnerName);

    circleCiAutoScalingGroup.addUserData(userDataScript);
}

Note that the auth token placeholder in the user data script is replaced with the value from an environment variable, and the runner name placeholder is replaced with stack property value.

Store credentials using AWS Secret Manager

The CircleCI token and the resource class name should be stored securely. You will use AWS Secret Manager to store the credentials instead of storing them as plaintext. Add this code snippet inside the constructor defined in the lib/circleci-self-hosted-runner-autoscaling-stack.ts file to create a new secret.

import {
    //update the existing import to add aws_secretsmanager
    aws_secretsmanager as secretsmanager,
    SecretValue,
  } from 'aws-cdk-lib';

constructor(scope: Construct, id: string, props ?: CircleciSelfHostedRunnerAutoscalingStackProps) {

    // add the following code snippet below the existing constructs
    const circleCISecret = new secretsmanager.Secret(this, 'CircleCiSelfHostedRunnerSecret', {
        secretName: 'circleci-self-hosted-runner-secret',
        secretObjectValue: {
            "resource_class": SecretValue.unsafePlainText(env.SELF_HOSTED_RUNNER_RESOURCE_CLASS),
            "circle_token": SecretValue.unsafePlainText(env.CIRCLECI_TOKEN),
        }
    });
}

Notice that you are fetching the credentials from CircleCI environment variables. The AWS Lambda function will fetch these credentials from the secrets manager before calling the CircleCI tasks API. You could also set the credentials as AWS Lambda environment variables, but anyone with access to the AWS console could view those values.

Create an AWS Lambda function

First, define an execution role for the AWS Lambda function. The execution role should have permission to GET the credentials from the secret manager and update the autoscaling count. Add this code snippet inside the constructor defined in the lib/circleci-self-hosted-runner-autoscaling-stack.ts file.

import {
    //update the existing import to add aws_iam
    aws_iam as iam,
} from 'aws-cdk-lib';

constructor(scope: Construct, id: string, props ?: CircleciSelfHostedRunnerAutoscalingStackProps) {

    // add the following code snippet below the existing constructs
    const lambdaPolicyDocument = new iam.PolicyDocument({
        statements: [
            new iam.PolicyStatement({
                resources: [circleCiAutoScalingGroup.autoScalingGroupArn],
                actions: ["autoscaling:UpdateAutoScalingGroup"],
            }),
            new iam.PolicyStatement({
                resources: [circleCISecret.secretArn],
                actions: ["secretsmanager:GetSecretValue"],
            })
        ],
    });

    const inferenceLambdaRole = new iam.Role(this, `CircleCIAutoScalingLambdaRole`, {
        assumedBy: new iam.ServicePrincipal("lambda.amazonaws.com"),
        description: "Role assumed by auto scaling lambda",
        inlinePolicies: {
            lambdaPolicy: lambdaPolicyDocument,
        },
        managedPolicies: [
            iam.ManagedPolicy.fromAwsManagedPolicyName("service-role/AWSLambdaBasicExecutionRole")
        ]
    });
}

To define an AWS Lambda function, add a CDK construct to create an AWS Lambda function inside the constructor defined in the lib/circleci-self-hosted-runner-autoscaling-stack.ts file.

import {
    //update the existing import to add aws_lambda and Duration
    aws_lambda as lambda,
    Duration,
} from 'aws-cdk-lib';

constructor(scope: Construct, id: string, props ?: CircleciSelfHostedRunnerAutoscalingStackProps) {

    // add the following code snippet below the existing constructs
    const autoScalingLambda = new lambda.Function(this, 'CircleCiSelfHostedRunnerAutoScalingLambda', {
        functionName: 'CircleCiSelfHostedRunnerAutoScalingLambda',
        code: lambda.Code.fromAsset('./lambda/auto-scaling-lambda/'),
        runtime: lambda.Runtime.NODEJS_14_X,
        handler: "index.handler",
        environment: {
            "SECRET_NAME": circleCISecret.secretName,
            "SECRET_REGION": props?.env?.region || 'us-west-2',
            "AUTO_SCALING_MAX": props!!.maxInstances,
            "AUTO_SCALING_GROUP_NAME": circleCiAutoScalingGroup.autoScalingGroupName,
            "AUTO_SCALING_GROUP_REGION": props?.env?.region || 'us-west-2'
        },
        timeout: Duration.minutes(1),
        role: inferenceLambdaRole
    });
}

Schedule the Lambda to run periodically

Finally, schedule the AWS Lambda function to be triggered every minute. You could choose a different frequency based on your expected workload. Add this code snippet inside the constructor defined in the lib/circleci-self-hosted-runner-autoscaling-stack.ts file.

import {
    //update the existing import to add aws_lambda and Duration
    aws_events as events,
    aws_events_targets as targets,
} from 'aws-cdk-lib';

constructor(scope: Construct, id: string, props ?: CircleciSelfHostedRunnerAutoscalingStackProps) {

    // add the following code snippet below the existing constructs
    const eventRule = new events.Rule(this, 'CircleCiLambdaSchedule', {
        schedule: events.Schedule.rate(Duration.minutes(1)),
    });

    eventRule.addTarget(new targets.LambdaFunction(autoScalingLambda))
}

Update the CDK app

The CircleciSelfHostedRunnerAutoscalingStack stack accepts a few parameters that need to be passed from the CDK app. Replace the code in the bin/circleci-self-hosted-runner-autoscaling.ts file with this code snippet:

import "source-map-support/register";
import * as cdk from "aws-cdk-lib";
import { CircleciSelfHostedRunnerAutoscalingStack } from "../lib/circleci-self-hosted-runner-autoscaling-stack";
const { env } = require("process");

const app = new cdk.App();
new CircleciSelfHostedRunnerAutoscalingStack(app, "CircleciSelfHostedRunnerAutoscalingStack", {
  maxInstances: "4",
  keypairName: env.AWS_KEYPAIR_NAME,
  runnerName: "aws-runner",
});

Notice that you are hardcoding the values for the max number of instances and the runner name. Also, the AWS_KEYPAIR_NAME is fetched from an environment variable.

Deploy the CDK stack

Now you can go ahead and deploy the application to an AWS account. Deploy the application manually first to make sure everything works before automating the deployments.

Note: Local deployment requires you to set up the AWS CLI, AWS CDK CLI, and a few environment variables on your machine. You can skip this section if you want to test your CDK stack using CircleCI directly.

Before deploying for the first time, you need to bootstrap the project using the cdk CLI. Bootstrapping the app provisions the resources that AWS CDK might require to deploy your application. Issue this command from the root of the project:

cdk bootstrap

Make sure that you have the AWS credentials configured on your system. Refer to the link in the prerequisite section for configuring AWS credentials. If the credentials are configured, CDK will use them automatically.

Next, deploy the application to the AWS account.

cdk deploy

Make sure you have the required environment variables set locally before deploying the stack. You need to set AWS_KEYPAIR_NAME, SELF_HOSTED_RUNNER_RESOURCE_CLASS, SELF_HOSTED_RUNNER_AUTH_TOKEN, and CIRCLECI_TOKEN to deploy the app.

Once you execute the command, you might be prompted to confirm the IAM role/policy changes applied to your account. Deployment should work successfully if your application is set up correctly.

Automate deployment using CircleCI

Well done! You can deploy the CDK application manually using the command line. Now you can automate the workflow so that the infrastructure changes can be packaged and deployed automatically every time you push code to the main branch. To automate the deployments:

1. Update .gitignore
2. Update NPM scripts
3. Add configuration script
4. Create a CircleCI project for the application
5. Set up environment variables

Update .gitignore

The code generated by the cdk init command contains a .gitignore file that ignores all .js files by default. Replace the contents of .gitignore with this code snippet:

!jest.config.js
*.d.ts
node_modules

# CDK asset staging directory
.cdk.staging
cdk.out

Update NPM scripts

Your CircleCI deployment configuration uses NPM scripts for executing the deploy and diff commands. Add these scripts to the root level package.json file:

// update the aws-cdk-lambda-circle-ci/package.json file with the following scripts
{
  ...
  "scripts": {
    ...
    // add the ci_diff and ci_deploy scripts
    "ci_diff": "cdk diff -c env=${ENV:-stg} 2>&1 | sed -r 's/\\x1B\\[([0-9]{1,2}(;[0-9]{1,2})?)?[mGK]//g' || true",
    "ci_deploy": "cdk deploy -c env=${ENV:-stg} --require-approval never"
  },
  ...
}

Add configuration script

Add a .circleci/config.yaml script in the project’s root (containing the configuration file for the CI pipeline). Add this code snippet to it:

version: 2.1

orbs:
  aws-cli: circleci/aws-cli@2.0.6
executors:
  default:
    docker:
      - image: "cimg/node:14.18.2"
    environment:
      AWS_REGION: "us-west-2"
jobs:
  build:
    executor: "default"
    steps:
      - aws-cli/setup:
          aws-access-key-id: AWS_ACCESS_KEY
          aws-secret-access-key: AWS_ACCESS_SECRET
          aws-region: AWS_REGION_NAME
      - checkout
      - run:
          name: "install_lambda_packages"
          command: |
            cd lambda/auto-scaling-lambda && npm install
            cd ../
      - run:
          name: "build"
          command: |
            npm install
            npm run build
      - run:
          name: "cdk_diff"
          command: |
            if [ -n "$CIRCLE_PULL_REQUEST" ]; then
              export ENV=stg
              if [ "${CIRCLE_BRANCH}" == "develop" ]; then
                export ENV=prd
              fi 
              pr_number=${CIRCLE_PULL_REQUEST##*/}
              block='```'
              diff=$(echo -e "cdk diff (env=${ENV})\n${block}\n$(npm run --silent ci_diff)\n${block}")
              data=$(jq -n --arg body "$diff" '{ body: $body }') # escape
              curl -X POST -H 'Content-Type:application/json' \
                -H 'Accept: application/vnd.github.v3+json' \
                -H "Authorization: token ${GITHUB_TOKEN}" \
                -d "$data" \
                "https://api.github.com/repos/${CIRCLE_PROJECT_USERNAME}/${CIRCLE_PROJECT_REPONAME}/issues/${pr_number}/comments"
            fi
      - run:
          name: "cdk_deploy"
          command: |
            if [ "${CIRCLE_BRANCH}" == "main" ]; then
              ENV=prd npm run ci_deploy
            elif [ "${CIRCLE_BRANCH}" == "develop" ]; then
              ENV=stg npm run ci_deploy
            fi

The CI script uses CircleCI’s aws-cli orb for setting AWS configuration, (the access key and the secret). The build job installs the packages, calculates the diff, and deploys the changes. The cdk_diff step executes only on pull requests and adds a comment on the PR summarizing the infrastructure changes.

The cdk_deploy command checks the branch and deploys on the prd or stg environment. Note that the cdk_deploy command executes the ci_deploy script defined in the package.json file.

Your pipeline configuration will take care of building, packaging, and deploying the CDK stack to the specified AWS account. Commit the changes and push them to the GitHub repository.

Note: Don’t forget to replace the AWS_REGION specified above your region if it differs.

Create a CircleCI project for the application

Set up the repository as a CircleCI project using the CircleCI console. On the CircleCI console, click Projects, search for the GitHub repo name and click Set Up Project for your project.

You will be prompted to add a new configuration file manually or use an existing one. Because you have already pushed the required configuration file to the codebase, select the Fastest option. Enter the name of the branch hosting your configuration file. Click Set Up Project to continue.

Completing the setup will trigger the pipeline. As expected, the pipeline will fail in its first run because you haven’t defined the environment variables.

Set up environment variables

Click Project settings from the project dashboard and go to the Environment variables tab. Click the Add environment variable button to add a new key value.

You need to add these environment variables:

• AWS_ACCESS_KEY is the access key obtained while creating AWS credentials.
• AWS_ACCESS_SECRET is the secret obtained while creating AWS credentials.
• AWS_REGION_NAME is the region where you wish to deploy your application.
• AWS_KEYPAIR_NAME is the name of the EC2 key pair created using the AWS console.
• SELF_HOSTED_RUNNER_RESOURCE_CLASS is the runner name you set while creating a self-hosted runner resource class.
• SELF_HOSTED_RUNNER_AUTH_TOKEN is the token obtained while creating a self-hosted runner resource class.
• CIRCLECI_TOKEN is the CircleCI personal access token. It is not the same as the CircleCI runner token.

Once the environment variables are configured, rerun the pipeline. This time it should build successfully.

Testing the Auto Scaling setup

Now that your autoscaling stack has deployed successfully, you can use it to run some jobs. To test the setup, use the example NodeJS app that you can clone using this link.

To run the jobs using your self-hosted runners, make sure the job config uses the self-hosted resource class you created earlier. Refer to this code snippet as an example for setting the resource_class:

version: 2.1
workflows:
  testing:
    jobs:
      - runner-test

jobs:
  runner-test: # this can be any name you choose
    machine: true
    resource_class: tutorial-gwp/auto-scaling-st ack
    steps:
      - checkout # checkout source code
      - run:
          name:
          command: npm test

Follow the steps from the previous section to create a CircleCI project for the NodeJS application. From the CircleCI project page, click Trigger pipeline to trigger a job for your application. You could click on it multiple times to have multiple jobs running simultaneously.

Go to the AWS EC2 console and note the new instances spinning up. The number of instances spinning up would equal the number of pending jobs.

If the number of pending jobs exceeds the maxCapacity of the AWS EC2 autoscaling group, only the maxCapacity number of instances will be spun up. Other pending jobs will be processed once one of these completes execution.

After a few minutes, you will notice that all the jobs were completed successfully.

Conclusion

In this tutorial, you learned how to autoscale self-hosted runners based on demand using AWS CDK. With AWS CDK, you can provision resources for your application using languages familiar to you. AWS CDK allows you to use logical statements and object-oriented techniques while defining your application. CircleCI’s self-hosted runners satisfy unique architecture requirements, privileged access, and control requirements. You can optimize costs and spin up additional resources based on demand by autoscaling for self-hosted runner resources.

Check out the complete source code used in this tutorial on GitHub. The GitHub project can also be used as a template if you are trying to define a similar stack.

The source code for the example NodeJS app can be found here.