CircleCI Server v3.x Installation Phase 1

Phase 1: Prerequisites

Install CircleCI server in 4 phases. There is a validation step at the end of each phase, allowing you to confirm success before moving to the next phase. Depending on your requirements, phases 3 and 4 may have multiple steps. This installation guide assumes you have already read the server 3.x overview.

In the following steps replace any items or credentials displayed between < > with your details.
Flow chart showing the installation flow for server 3.x
Figure 1. Installation Experience Flow Chart

Install required software

Download and install the following software before continuing:

Tool Version Used for

Terraform

0.15.4 or greater

Infrastructure Management

kubectl

1.19 or greater

Kubernetes CLI

Helm

3.4.0 or greater

Kubernetes Package Management

Kots

1.47.3 or greater

Replicated Kubernetes Application Management

Create a Kubernetes cluster

CircleCI server installs into an existing Kubernetes cluster. The application uses a large number of resources. Depending on your usage, your Kubernetes cluster should meet the following requirements:

Number of daily active CircleCI users Minimum Nodes Total CPU Total RAM NIC speed

< 500

3

12 cores

32 GB

1 Gbps

500+

3

48 cores

240 GB

10 Gbps

Creating a Kubernetes cluster is your responsibility. Please note:

  • Your cluster must have outbound access to pull Docker containers and verify your license. If you do not want to provide open outbound access, see our list of ports that will need access.

  • You must have appropriate permissions to list, create, edit and delete pods in your cluster. Run this to verify your permissions:

    kubectl auth can-i <list|create|edit|delete> pods
  • There are no requirements regarding VPC setup or disk size for your cluster. It is recommended that you set up a new VPC rather than use an existing one.

EKS

You can learn more about creating an Amazon EKS cluster here. We recommend using eksctl to create your cluster, which will create a VPC and select the proper security groups for you.

  1. Install and configure the AWS CLI for your AWS account.

  2. Install eksctrl.

  3. Create your cluster by running the following (Cloud formation with eksctl and EKS can take upwards of 20 minutes to complete):

    eksctl create cluster --name=circleci-server --nodes 4 --node-type m5.xlarge
  4. Once the cluster has been created, you can use the following command to configure kubectl access:

    eksctl utils write-kubeconfig --name circleci-server
You may see the following error AWS STS Access - cannot get role ARN for current session: InvalidClientTokenID. This means your AWS credentials are invalid, or your IAM user does not have permission to create an EKS cluster. Proper IAM permissions are necessary in order to use eksctl. See the AWS documentation regarding IAM permissions.

GKE

You can learn more about creating a GKE cluster here.

Do not use Autopilot cluster. CircleCI requires functionality that is not supported by GKE Autopilot.
  1. Install and configure the GCP CLI for your GCP account. This includes creating a Google Project, which will be required to create a cluster within your project. When you create your project make sure you also enabled API access. If you do not enable API access, the command we will run next, to create your cluster, will fail.

  2. Create your cluster by entering running the following:

    gcloud container clusters create circleci-server --project <YOUR_GOOGLE_CLOUD_PROJECT_ID> --region europe-west1 --num-nodes 3 --machine-type n1-standard-4
  3. Configure` kubectl` with your your credentials gcloud credentials:

    gcloud container clusters get-credentials circleci-server --region europe-west1
  4. Verify your cluster:

    kubectl cluster-info
  5. Create a service account for this cluster:

    gcloud iam service-accounts create <YOUR_SERVICE_ACCOUNT_ID> --description="<YOUR_SERVICE_ACCOUNT_DISPLAY_NAME>"  --display-name="<YOUR_SERVICE_ACCOUNT_DISPLAY_NAME>"
  6. Get the credentials for the service account:

    gcloud iam service-accounts keys create <PATH_TO_STORE_CREDENTIALS> --iam-account <SERVICE_ACCOUNT_ID>@<YOUR_GOOGLE_CLOUD_PROJECT_ID>.iam.gserviceaccount.com

Create a new GitHub OAuth app

Registering and setting up a new GitHub OAuth app for CircleCI server allows for authorization control to your server installation using GitHub OAuth and for updates to GitHub projects/repos using build status information.

  1. In your browser navigate to your GitHub instance > Settings > Developer Settings > OAuth Apps and click the New OAuth App button.

    Screenshot showing setting up a new OAuth app
    Figure 2. New GitHub OAuth App
  2. Complete the following fields based on your planned installation:

    • Homepage URL: The URL of your planned CircleCI installation.

    • Authorization callback URL: The authorization callback URL will be the URL of your planned CircleCI installation followed by /auth/github

  3. Once completed you will be shown the Client ID. Select Generate a new Client Secret to generate a Client Secret for your new OAuth App. You will need these values when you configure CircleCI server.

    Screenshot showing GitHub Client ID
    Figure 3. Client ID and Secret
If using GitHub Enterprise, you will also need a personal access token and the domain name of your GitHub Enterprise instance.

Frontend TLS certificates

By default, CircleCI server will create self-signed certificates to get you started. In production, you should supply a certificate from a trusted certificate authority. The LetsEncrypt certificate authority, for example, can issue a certificate for free using their certbot tool. In the sections below we cover using Google Cloud DNS and AWS Route53.

Google Cloud DNS

  1. If you host your DNS on Google Cloud you will need the certbot-dns-google plugin installed. You can install the plugin with the following command:

    pip3 install certbot-dns-google
  2. Then, the following commands will provision a certification for your installation:

    certbot certonly --dns-google --dns-google-credentials <PATH_TO_CREDENTIALS> -d "<CIRCLECI_SERVER_DOMAIN>" -d "app.<CIRCLECI_SERVER_DOMAIN>"

AWS Route53

  1. If you are using AWS Route53 for DNS you will need the certbot-route53 plugin installed. You can install the plugin with the following command:

    pip3 install certbot-dns-route53
  2. Then execute this example to create a private key and certificate (including intermediate certificates) locally in /etc/letsencrypt/live/<CIRCLECI_SERVER_DOMAIN>:

    certbot certonly --dns-route53 -d "<CIRCLECI_SERVER_DOMAIN>" -d "app.<CIRCLECI_SERVER_DOMAIN>"

You will need these certificates later, and they can be retrieved locally with the following commands:

ls -l /etc/letsencrypt/live/<CIRCLECI_SERVER_DOMAIN>
cat /etc/letsencrypt/live/<CIRCLECI_SERVER_DOMAIN>/fullchain.pem
cat /etc/letsencrypt/live/<CIRCLECI_SERVER_DOMAIN>/privkey.pem
It is important that your certificate contains both your domain and the app.* subdomain as subjects. For example, if you host your installation at server.example.com, your certificate must cover app.server.example.com and server.example.com

Encryption/signing keys

These keysets are used to encrypt and sign artifacts generated by CircleCI. You will need these values to configure server.

Store these values securely. If they are lost, job history and artifacts will not be recoverable.

Artifact signing key

To generate, run the following:

docker run circleci/server-keysets:latest generate signing -a stdout

Encryption signing key

To generate, run the following:

docker run circleci/server-keysets:latest generate encryption -a stdout

Object storage and permissions

Server 3.x hosts build artifacts, test results, and other state object storage. We support the following:

While any S3 compatible object storage may work, we test and support AWS S3 and Minio. For object storage providers that do not support S3 API, such as Azure blob storage, we recommend using Minio Gateway.

Please choose the option that best suits your needs. A Storage Bucket Name is required, in addition to the fields listed below, depending on whether you are using AWS or GCP. Ensure the bucket name you provide exists in your chosen object storage provider before proceeding.

If you are installing behind a proxy, object storage should be behind this proxy also, otherwise proxy details will need to be supplied at the job level within every project .circleci/config.yml to allow artifacts, test results, cache save and restore, and workspaces to work. For more information see the Configuring a Proxy guide.

Create an S3 storage bucket

You will need the following details when you configure CircleCI server.

  • Storage Bucket Name - The bucket name to be used for server.

  • Access Key ID - Access Key ID for S3 bucket access.

  • Secret Key - Secret Key for S3 bucket access.

  • AWS S3 Region - AWS region of bucket if your provider is AWS. You will either have an AWS region or S3 Endpoint depending on your specific setup.

  • S3 Endpoint - API endpoint of S3 storage provider, when your storage provider is not Amazon S3.

Step 1: Create AWS S3 Bucket
aws s3api create-bucket \
    --bucket <YOUR_BUCKET_NAME> \
    --region <YOUR_REGION> \
    --create-bucket-configuration LocationConstraint=<YOUR_REGION>
us-east-1 does not support a LocationConstraint. If your region is us-east-1, omit the bucket configuration
Step 2: Create an IAM user for CircleCI server
aws iam create-user --user-name circleci-server
Step 3: create a policy document "policy.json" with the following content
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:*"
      ],
      "Resource": [
        "arn:aws:s3:::<YOUR_BUCKET_NAME>",
        "arn:aws:s3:::<YOUR_BUCKET_NAME>/*"
      ]
    }
  ]
}
Step 4: Attach policy to user
aws iam put-user-policy \
  --user-name circleci-server \
  --policy-name circleci-server \
  --policy-document file://policy.json
Step 5: Create Access Key for user circleci-server
You will need this when you configure your server installation later.
aws iam create-access-key --user-name circleci-server

The result should look like this:

{
  "AccessKey": {
        "UserName": "circleci-server",
        "Status": "Active",
        "CreateDate": "2017-07-31T22:24:41.576Z",
        "SecretAccessKey": <AWS_SECRET_ACCESS_KEY>,
        "AccessKeyId": <AWS_ACCESS_KEY_ID>
  }
}

Create a Google Cloud storage bucket

You will need the following details when you configure CircleCI server.

  • Storage Bucket Name - The bucket used for server.

  • Service Account JSON - A JSON format key of the Service Account to use for bucket access.

A dedicated service account is recommended. Add to it the Storage Object Admin role, with a condition on the resource name limiting access to only the bucket specified above. For example, enter the following into the Google’s Condition Editor of the IAM console:

Use startsWith and prefix the bucket name with projects/_/buckets/.
resource.name.startsWith("projects/_/buckets/<YOUR_BUCKET_NAME>")
Step 1: Create a GCP bucket

If your server installation runs within a GKE cluster, ensure that your current IAM user is a cluster admin for this cluster, as RBAC (role-based access control) objects need to be created. More information can be found in the GKE documentation.

gsutil mb gs://circleci-server-bucket
Step 2: Create a Service Account
gcloud iam service-accounts create circleci-server --display-name "circleci-server service account"

You will need the email for the service account in the next step, run the following to find it:

gcloud iam service-accounts list \
  --filter="displayName:circleci-server account" \
  --format 'value(email)'
Step 3: Grant Permissions to Service Account
gcloud iam roles create circleci_server \
    --project <PROJECT_ID> \
    --title "CircleCI Server" \
    --permissions \ compute.disks.get,compute.disks.create,compute.disks.createSnapshot,compute.snapshots.get,compute.snapshots.create,compute.snapshots.useReadOnly,compute.snapshots.delete,compute.zones.get
gcloud projects add-iam-policy-binding <PROJECT_ID> \
    --member serviceAccount:<SERVICE_ACCOUNT_EMAIL> \
    --role projects/<PROJECT_ID>/roles/circleci_server
gsutil iam ch serviceAccount:<SERVICE_ACCOUNT_EMAIL>:objectAdmin gs://circleci-server-bucket
Step 4: JSON Key File

After running the following, you should have a file named circleci-server-keyfile in your local working directory. You will need this when you configure your server installation.

gcloud iam service-accounts keys create circleci-server-keyfile \
    --iam-account <SERVICE_ACCOUNT_EMAIL>


Help make this document better

This guide, as well as the rest of our docs, are open-source and available on GitHub. We welcome your contributions.