CircleCI Server v3.x Installation Phase 3

Before you begin with the CircleCI server v3.x build services installation phase, ensure you have run through Phase 1 – Prerequisites and Phase 2 - Core services installation.

In the following steps replace any items or credentials displayed between < > with your details.

Phase 3: Build services installation

Output Processor

Overview

Output processor is responsible for handling the output from Nomad clients. It is a key service to scale if you find your system slowing down. We recommend increasing the output processor replica set to scale the service up to meet demand.

Access the KOTS admin console. Get to the KOTS admin console by running the following, substituting your namespace: kubectl kots admin-console -n <YOUR_CIRCLECI_NAMESPACE>

Locate and enter the following in Settings:

  1. Output Processor Load Balancer (required) - The following command will provide the IP address of the service

    kubectl get service output-processor --namespace=<YOUR_CIRCLECI_NAMESPACE>
  2. Save your configuration. You will deploy and validate your setup after you complete Nomad client setup.

Nomad Clients

Overview

As mentioned in the Overview, Nomad is a workload orchestration tool that CircleCI uses to schedule (via Nomad Server) and run (via Nomad Clients) CircleCI jobs.

Nomad clients are installed outside of the Kubernetes cluster, while their control plane (Nomad Server) is installed within the cluster. The communication between your Nomad Clients and the nomad control plane are secured with mTLS. The mTLS certificate, private key, and certificate authority will be output after you complete the Nomad Clients installation.

Once completed you will be able to update your CircleCI server configuration so your Nomad control plane is able to communicate with your Nomad Clients.

Cluster Creation with Terraform

CircleCI curates Terraform modules to help install Nomad clients in your cloud provider of choice. You can browse the modules in our public repository including example Terraform config files (man.tf) for both AWS and GKEs for main.tf. Some information about your cluster and server installation is required to complete your main.tf. How to get this information is described in the following sections.

AWS

You will need some information about your cluster and server installation to complete the required fields for the terraform configuration file (main.tf). A full example as well as a full list of variables can be found here.

  • Server_endpoint - You will need to know the Nomad Server endpoint, which is the external IP address of the nomad-server-external Loadbalancer. You can get this information with the following command:

    kubectl get service nomad-server-external --namespace=<YOUR_CIRCLECI_NAMESPACE>
  • Subnet ID (subnet), VPC ID (vpcId), and DNS server (dns_server) of your cluster. Run the following command to get the cluster VPC ID (vpcId), CIDR block (serviceIpv4Cidr), and subnets (subnetIds):

    aws eks describe-cluster --name=<YOUR_CLUSTER_NAME>

    This will return something similar to the following:

    {...
    "resourcesVpcConfig": {
        "subnetIds": [
            "subnet-033a9fb4be69",
            "subnet-04e89f9eef89",
            "subnet-02907d9f35dd",
            "subnet-0fbc63006c5f",
            "subnet-0d683b6f6ba8",
            "subnet-079d0ca04301"
        ],
        "clusterSecurityGroupId": "sg-022c1b544e574",
        "vpcId": "vpc-02fdfff4c",
        "endpointPublicAccess": true,
        "endpointPrivateAccess": false
    ...
    "kubernetesNetworkConfig": {
                "serviceIpv4Cidr": "10.100.0.0/16"
            },
    ...
    }

    Then, using the VPCID you just found, run the following command to get the CIDR Block for your cluster. For AWS, the DNS Server is the third IP in your CIDR block (CidrBlock), for example your CIDR block might be 10.100.0.0/16 so the third IP would be 10.100.0.2.

    aws ec2 describe-vpcs --filters Name=vpc-id,Values=<YOUR_VPCID>

    This will return something like the following:

    {...
    "CidrBlock": "192.168.0.0/16",
    "DhcpOptionsId": "dopt-9cff",
    "State": "available",
    "VpcId": "vpc-02fdfff4c"
    ...}

Once you have filled in the appropriate information you can deploy your nomad clients by running the following from within the directory of the main.tf file.

terraform init
terraform plan
terraform apply

After Terraform is done spinning up the Nomad client(s), it will output the certificates and keys needed for configuring the Nomad control plane in CircleCI server. Make sure to copy them somewhere safe. The apply process usually only takes a minute.

GKE

You will need the IP address of the Nomad control plane (Nomad Server), which was created when you deployed CircleCI Server. You can get the IP address by issuing the following command:

kubectl get service nomad-server-external --namespace=<YOUR_CIRCLECI_NAMESPACE>

You will also need the following information:

  • The GPC Project you want to run nomad clients in.

  • The GPC Zone you want to run nomad clients in.

  • The GPC Region you want to run nomad clients in.

  • The GPC Network you want to run nomad clients in.

  • The ID of the GPC subnet you want to run nomad clients in.

You can copy the following example to your local environment and fill in the appropriate information for your specific setup. Once you have filled in the appropriate information you can deploy your nomad clients by running.

terraform init
terraform plan
terraform apply

After Terraform is done spinning up the Nomad client(s), it will output the certificates and key needed for configuring the Nomad control plane in CircleCI server. Make sure to copy them somewhere safe.

Configure and Deploy

Now that you have successfully deployed your Nomad clients, you can configure CircleCI Server and the Nomad control plane. Access the KOTS admin console. Get to the KOTS admin console by running the following, substituting your namespace: kubectl kots admin-console -n <YOUR_CIRCLECI_NAMESPACE>

Enter the following in Settings:

  • Nomad Load Balancer (required)

    kubectl get service nomad-server-external --namespace=<YOUR_CIRCLECI_NAMESPACE>
  • Nomad Server Certificate (required) - Provided in the output from terraform apply

  • Nomad Server Private Key (required) - Provided in the output from terraform apply

  • Nomad Server Certificate Authority (CA) Certificate (required) - Provided in the output from terraform apply

Click the Save config button to update your installation and re-deploy server.

Nomad Clients Validation

CircleCI has created a project called realitycheck which allows you to test your Server installation. We are going to follow the project so we can verify that the system is working as expected. As you continue through the next phase, sections of realitycheck will move from red to green.

To run realitycheck you will need to clone the repository. Depending on your Github setup you can do one of the following.

Github Cloud
git clone -b server-3.0 https://github.com/circleci/realitycheck.git
Github Enterprise
git clone -b server-3.0 https://github.com/circleci/realitycheck.git
git remote set-url origin <YOUR_GH_REPO_URL>
git push

Once you have successfully cloned the repository you can follow it from within your CircleCI server installation. You will need to set the following variables. For full instructions please see the repository readme.

Table 1. Environmental Variables
Name Value

CIRCLE_HOSTNAME

<YOUR_CIRCLECI_INSTALLATION_URL>

CIRCLE_TOKEN

<YOUR_CIRCLECI_API_TOKEN>

Table 2. Contexts
Name Environmental Variable Key

Environmental Variable Value

org-global

CONTEXT_END_TO_END_TEST_VAR

Leave blank

individual-local

MULTI_CONTEXT_END_TO_END_VAR

Once you have configured the environmental variables and contexts, rerun the realitycheck tests. You should see the features and resource jobs complete successfully. Your test results should look something like the following:

Screenshot showing the realitycheck project building in the CircleCI app

VM service

VM service configures VM and remote docker jobs. You can configure a number of options for VM service, such as scaling rules. VM service is unique to EKS and GKE installations because it specifically relies on features of these cloud providers.

EKS

  1. Get the Information Needed to Create Security Groups

    The following will return your VPC ID (vpcId), CIDR Block (serviceIpv4Cidr), Cluster Security Group ID (clusterSecurityGroupId) and Cluster ARN (arn) values, which you will need throughout this section:

    aws eks describe-cluster --name=<your-cluster-name>
  2. Create a security group

    Run the following commands to create a security group for VM service.

    aws ec2 create-security-group --vpc-id "<YOUR_VPCID>" --description "CircleCI VM Service security group" --group-name "circleci-vm-service-sg"

    This will output a GroupID to be used in the next steps:

    {
        "GroupId": "sg-0cd93e7b30608b4fc"
    }
  3. Apply security group Nomad

    Use the security group you just created and CIDR block values to apply the security group to the following:

    aws ec2 authorize-security-group-ingress --group-id "<YOUR_GroupId>" --protocol tcp --port 22 --cidr "<YOUR_serviceIpv4Cidr>"
    aws ec2 authorize-security-group-ingress --group-id "<YOUR_GroupId>" --protocol tcp --port 2376 --cidr "<YOUR_serviceIpv4Cidr>"
    If you created your Nomad Clients in a different subnet from CircleCI server, you will need to rerun the above two commands with each subnet CIDR.
  4. Apply the Security Group for SSH

    Run the following command to apply the security group rules so users can SSH into their jobs:

    aws ec2 authorize-security-group-ingress --group-id "<YOUR_GroupId>" --protocol tcp --port 54782
  5. Create user

    Create a new user with programmatic access:

    aws iam create-user --user-name circleci-server-vm-service
  6. Create Policy

    Create a policy.json file with the following content. You should fill in Cluster Security Group ID (clusterSecurityGroupId) and Cluster ARN (arn) below.

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Action": "ec2:RunInstances",
          "Effect": "Allow",
          "Resource": [
            "arn:aws:ec2:*::image/*",
            "arn:aws:ec2:*::snapshot/*",
            "arn:aws:ec2:*:*:key-pair/*",
            "arn:aws:ec2:*:*:launch-template/*",
            "arn:aws:ec2:*:*:network-interface/*",
            "arn:aws:ec2:*:*:placement-group/*",
            "arn:aws:ec2:*:*:volume/*",
            "arn:aws:ec2:*:*:subnet/*",
            "arn:aws:ec2:*:*:security-group/<YOUR_clusterSecurityGroupID>"
          ]
        },
        {
          "Action": "ec2:RunInstances",
          "Effect": "Allow",
          "Resource": "arn:aws:ec2:*:*:instance/*",
          "Condition": {
            "StringEquals": {
              "aws:RequestTag/ManagedBy": "circleci-server-vm-service"
            }
          }
        },
        {
          "Action": [
            "ec2:CreateVolume"
          ],
          "Effect": "Allow",
          "Resource": [
            "arn:aws:ec2:*:*:volume/*"
          ],
          "Condition": {
            "StringEquals": {
              "aws:RequestTag/ManagedBy": "circleci-server-vm-service"
            }
          }
        },
        {
          "Action": [
            "ec2:Describe*"
          ],
          "Effect": "Allow",
          "Resource": "*"
        },
        {
          "Effect": "Allow",
          "Action": [
            "ec2:CreateTags"
          ],
          "Resource": "arn:aws:ec2:*:*:*/*",
          "Condition": {
            "StringEquals": {
              "ec2:CreateAction" : "CreateVolume"
            }
          }
        },
        {
          "Effect": "Allow",
          "Action": [
            "ec2:CreateTags"
          ],
          "Resource": "arn:aws:ec2:*:*:*/*",
          "Condition": {
            "StringEquals": {
              "ec2:CreateAction" : "RunInstances"
            }
          }
        },
        {
          "Action": [
            "ec2:CreateTags",
            "ec2:StartInstances",
            "ec2:StopInstances",
            "ec2:TerminateInstances",
            "ec2:AttachVolume",
            "ec2:DetachVolume",
            "ec2:DeleteVolume"
          ],
          "Effect": "Allow",
          "Resource": "arn:aws:ec2:*:*:*/*",
          "Condition": {
            "StringEquals": {
              "ec2:ResourceTag/ManagedBy": "circleci-server-vm-service"
            }
          }
        },
        {
          "Action": [
            "ec2:RunInstances",
            "ec2:StartInstances",
            "ec2:StopInstances",
            "ec2:TerminateInstances"
          ],
          "Effect": "Allow",
          "Resource": "arn:aws:ec2:*:*:subnet/*",
          "Condition": {
            "StringEquals": {
              "ec2:Vpc": "<YOUR_arn>"
            }
          }
        }
      ]
    }
  7. Attach Policy to User

    Once you have created the policy.json file attach it to an IAM policy and created user.

    aws iam put-user-policy --user-name circleci-server-vm-service --policy-name circleci-server-vm-service --policy-document file://policy.json
  8. Create an access key and secret for the user

    If you have not already, you will need an access key and secret for the circleci-server-vm-service user. You can create that by running the following command:

    aws iam create-access-key --user-name circleci-server-vm-service
  9. Configure Server

    Configure VM Service through the KOTs admin console. The following fields need to be completed for VM service to operate correctly.

    • AWS Region (required) - This is the region the application is in.

    • Subnets (required) - Choose a subnet (public or private) where the VMs should be deployed. If you haven’t created a unique subnet you can use the subnet of the cluster. Note that all subnets must be in the same availability zone.

    • Security Group ID (required) - This is the security group that will be attached to the VMs. It was created previously.

    • AWS IAM Access Key ID (required) - AWS Access Key ID for EC2 access.

    • AWS IAM Secret Key (required) - IAM Secret Key for EC2 access.

    • AWS Windows AMI ID (optional) - If you require Windows builders, you can supply an AMI ID for them here.

Once you have configured the fields, save your config and deploy your updated application.

GKE

You will need additional information about your cluster to complete the next section. Run the following:

gcloud container clusters describe

This command will return something like the following, which will include network, region and other details that you will need to complete the next section:

addonsConfig:
  gcePersistentDiskCsiDriverConfig:
    enabled: true
  kubernetesDashboard:
    disabled: true
  networkPolicyConfig:
    disabled: true
clusterIpv4Cidr: 10.100.0.0/14
createTime: '2021-08-20T21:46:18+00:00'
currentMasterVersion: 1.20.8-gke.900
currentNodeCount: 3
currentNodeVersion: 1.20.8-gke.900
databaseEncryption:
  1. Create firewall rules

    Run the following commands to create a firewall rules for VM service in GKE:

    gcloud compute firewall-rules create "circleci-vm-service-internal-nomad-fw" --network "<network>" --action allow --source-ranges "0.0.0.0/0" --rules "TCP:22,TCP:2376"
    You can find the Nomad clients CIDR based on the region by referring to the table here if you have used auto-mode.
    gcloud compute firewall-rules create "circleci-vm-service-internal-k8s-fw" --network "<network>" --action allow --source-ranges "<clusterIpv4Cidr>" --rules "TCP:22,TCP:2376"
    gcloud compute firewall-rules create "circleci-vm-service-external-fw" --network "<network>" --action allow --rules "TCP:54782"
  2. Create user

    We recommend you create a unique service account used exclusively by VM Service. The Compute Instance Admin (Beta) role is broad enough to allow VM Service to operate. If you wish to make permissions more granular, you can use the Compute Instance Admin (beta) role documentation as reference.

    gcloud iam service-accounts create circleci-server-vm --display-name "circleci-server-vm service account"
  3. Get the service account email address

    gcloud iam service-accounts list --filter="displayName:circleci-server-vm service account" --format 'value(email)'
  4. Apply role to service account

    Apply the Compute Instance Admin (Beta) role to the service account.

    gcloud projects add-iam-policy-binding <YOUR_PROJECT_ID> --member serviceAccount:<YOUR_SERVICE_ACCOUNT_EMAIL> --role roles/compute.instanceAdmin --condition=None

    And

    gcloud projects add-iam-policy-binding <YOUR_PROJECT_ID> --member serviceAccount:<YOUR_SERVICE_ACCOUNT_EMAIL> --role roles/iam.serviceAccountUser --condition=None
  5. Get JSON Key File

    After running the following, you should have a file named circleci-server-vm-keyfile in your local working directory. You will need this when you configure your server installation.

    gcloud iam service-accounts keys create circleci-server-vm-keyfile --iam-account <YOUR_SERVICE_ACCOUNT_EMAIL>
  6. Configure Server

    Configure VM service through the KOTS admin console:

    • VM Service Load Balancer (required) This can be found using the following command:

      kubectl get service vm-service --namespace=<YOUR_CIRCLECI_NAMESPACE>
    • GCP project ID (required) - Name of the GCP project the cluster resides.

    • GCP Zone (required) - GCP zone the virtual machines instances should be created in for example “us-east1-b”.

    • GCP VPC Network (required) - Name of the VPC Network.

    • GCP VPC Subnet (optional) - Name of the VPC Subnet. If using auto-subnetting, leave this field blank.

    • GCP Service Account JSON Key File (required) - Copy and paste the contents of your service account JSON file.

    • GCP Windows Image (optional) - Name of the image used for Windows builds. Leave this field blank if you do not require them.

Click the Save config button to update your installation and re-deploy server.

Additional VM Service Configuration

  • Number of <VM type> VMs to keep prescaled (optional) - By default, this field is set to 0 which will create and provision instances of a resource type on demand. You have the option of preallocating up to 5 instances per resource type. Preallocating instances lowers the start time allowing for faster machine and remote_docker builds.

    that preallocated instances are always running and could potentially increase costs. Decreasing this number may also take up to 24 hours for changes to take effect. You have the option of terminating those instances manually, if required.

VM Service Validation

Once you have configured and deployed CircleCI server you should validate that VM Service is operational. You can re-run the reality checker project within your CircleCI installation and you should see the VM Service Jobs complete with green. At this point all tests should pass with green.

Runner

Overview

CircleCI runner does not require any additional server configuration. Server ships ready to work with runner. However, you do need to create a runner and configure the runner agent to be aware your server installation. For complete instructions for setting up runner see the runner documentation.

Runner requires a namespace per organization. Server can have many organizations. If your company has multiple organizations within your CircleCI installation you will need to set up a runner namespace for each organization within your server installation.


Help make this document better

This guide, as well as the rest of our docs, are open-source and available on GitHub. We welcome your contributions.