CircleCI Server v3.x Installation Phase 1

Server v3.x

Server Admin

Phase 1: Prerequisites

CircleCI server v3.x is installed in 4 phases. There is a validation step at the end of each phase, allowing you to confirm success before moving to the next phase. Depending on your requirements, phases 3 and 4 may include multiple steps. This installation guide assumes you have already read the CircleCI Server v3.x Overview.

Flow chart showing the installation flow for server 3.x with phase 1 highlighted

Figure 1. Installation Experience Flow Chart Phase 1

In the following sections, replace any items or credentials displayed between < > with your details.

Install required software

Download and install the following software before continuing:

Tool	Version	Used for	Notes
Terraform	0.15.4 or greater	Infrastructure Management
kubectl	1.19 or greater	Kubernetes CLI
Helm	3.9.2 or greater	Kubernetes Package Management
KOTS: Mac or Linux.	1.65.0 *	Replicated Kubernetes Application Management. KOTS is a `kubectl` plugin.	Once you have extracted `kots` from the tar.gz (`tar zxvf kots_linux_amd64.tar.gz`), run `sudo mv kots /usr/local/bin/kubectl-kots` to install it. Mac users will need to grant a security exception.
Velero CLI	Latest	Backup and restore capability	See Velero’s supported providers documentation for further information.

* Please take note of the supported KOTS versions for your Kubernetes cluster. KOTS version compatibility

AWS required software

AWS CLI

GCP required software

gcloud and gsutil. You can install and set-up these tools up by installing Google Cloud SDK. For further information refer to the Google Cloud SDK docs.

S3 compatible storage required software

Install and configure MinIO CLI for your storage provider.

Create a Kubernetes cluster

CircleCI server installs into an existing Kubernetes cluster. The application uses a large number of resources. Depending on your usage, your Kubernetes cluster should meet the following requirements:

Number of daily active CircleCI users	Minimum Nodes	Total CPU	Total RAM	NIC speed
< 500	3	12 cores	32 GB	1 Gbps
500+	3	48 cores	240 GB	10 Gbps

Supported Kubernetes versions:

CircleCI Version	Kubernetes Version
3.0.0 - 3.2.1	< 1.21
3.2.2 - 3.3.0	1.16 - 1.21
3.4.0 - 3.4.6	1.16 - 1.23

Creating a Kubernetes cluster is your responsibility. Please note:

Your cluster must have outbound access to pull Docker containers and verify your license. If you do not want to provide open outbound access, see our list of ports that need access.
You must have appropriate permissions to list, create, edit, and delete pods in your cluster. Run this command to verify your permissions:
```
kubectl auth can-i <list|create|edit|delete> pods
```
There are no requirements regarding VPC setup or disk size for your cluster. It is recommended that you set up a new VPC rather than use an existing one.

EKS

You can learn more about creating an Amazon EKS cluster here. We recommend using eksctl to create your cluster, which creates a VPC and selects the proper security groups for you.

Install and configure the AWS CLI for your AWS account.
Install eksctl.
Create your cluster by running the following (Cloud formation with eksctl and EKS can take more than 20 minutes to complete):
```
eksctl create cluster --name=circleci-server --nodes 4 --node-type m5.xlarge
```
Once the cluster has been created, you can use the following command to configure kubectl access:
```
eksctl utils write-kubeconfig --cluster circleci-server
```

You may see the following error AWS STS Access - cannot get role ARN for current session: InvalidClientTokenID. This means your AWS credentials are invalid, or your IAM user does not have permission to create an EKS cluster. Proper IAM permissions are necessary to use eksctl. See the AWS documentation on IAM permissions.

GKE

You can learn more about creating a GKE cluster here.

Do not use Autopilot cluster. CircleCI requires functionality that is not supported by GKE Autopilot.

Install and configure the GCP CLI for your GCP account. This includes creating a Google Project, which will be required to create a cluster within your project.

When you create your project, make sure you also enable API access. If you do not enable API access, the command we will run next (to create your cluster) will fail.

Create your cluster by running the following command:

gcloud container clusters create circleci-server --project <YOUR_GOOGLE_CLOUD_PROJECT_ID> --region europe-west1 --num-nodes 3 --machine-type n1-standard-4

Configure kubectl with your your gcloud credentials:

gcloud container clusters get-credentials circleci-server --region europe-west1

Verify your cluster:
```
kubectl cluster-info
```

Create a service account for this cluster:

gcloud iam service-accounts create <YOUR_SERVICE_ACCOUNT_ID> --description="<YOUR_SERVICE_ACCOUNT_DESCRIPTION>"  --display-name="<YOUR_SERVICE_ACCOUNT_DISPLAY_NAME>"

Get the credentials for the service account:

gcloud iam service-accounts keys create <PATH_TO_STORE_CREDENTIALS> --iam-account <SERVICE_ACCOUNT_ID>@<YOUR_GOOGLE_CLOUD_PROJECT_ID>.iam.gserviceaccount.com

Enable Workload Identities in GKE (optional)

Workload Identities for GKE allow workloads/pods in your GKE cluster to impersonate IAM service accounts to access Google Cloud services without using static service account credentials. In order to use Workload Identities you must enable them on your GKE cluster.

Enable Workload Identity on existing cluster

  gcloud container clusters update "<CLUSTER_NAME>" \
    --region="<REGION>" \
    --workload-pool="<PROJECT_ID>.svc.id.goog"

Get node pools of existing GKE cluster

  gcloud container node-pools list --cluster "<CLUSTER_NAME>" --region "<REGION>"

Update existing node pools

  gcloud container node-pools update "<NODEPOOL_NAME>" \
    --cluster="<CLUSTER_NAME>" \
    --workload-metadata="GKE_METADATA" \
    --region="<REGION>"

You must repeat Step 3 for all the existing node pools. Follow these links for steps to enable Workload Identity for your Kubernetes service accounts: Nomad Autoscaler, VM and Object-Storage

Create a new GitHub OAuth app

If GitHub Enterprise and CircleCI server are not on the same domain, then images and icons from GHE will fail to load in the CircleCI web app.

Registering and setting up a new GitHub OAuth app for CircleCI server allows for authorization control to your server installation using GitHub OAuth and for updates to GitHub projects/repos using build status information.

In your browser, navigate to your GitHub instance > Settings > Developer Settings > OAuth Apps and click the New OAuth App button.

Figure 2. New GitHub OAuth App
Complete the following fields, based on your planned installation:
- Homepage URL: The URL of your planned CircleCI installation.
- Authorization callback URL: The authorization callback URL is the URL of your planned CircleCI installation followed by /auth/github
Once completed, you will be shown the Client ID. Select Generate a new Client Secret to generate a Client Secret for your new OAuth App. You need these values when you configure CircleCI server.

Figure 3. Client ID and Secret

If using GitHub Enterprise, you also need a personal access token and the domain name of your GitHub Enterprise instance.

Frontend TLS certificates

By default, CircleCI server creates self-signed certificates to get you started. In production, you should supply a certificate from a trusted certificate authority. The Let’s Encrypt certificate authority, for example, can issue a free certificate using their certbot tool. The sections below cover using Google Cloud DNS and AWS Route 53.

AWS Route 53

If you are using AWS Route 53 for DNS, you need the certbot-route53 plugin installed. You can install the plugin with the following command:
```
pip3 install certbot-dns-route53
```
Then execute this example to create a private key and certificate (including intermediate certificates) locally in /etc/letsencrypt/live/<CIRCLECI_SERVER_DOMAIN>:
```
certbot certonly --dns-route53 -d "<CIRCLECI_SERVER_DOMAIN>" -d "app.<CIRCLECI_SERVER_DOMAIN>"
```

It is important that your certificate contains both your domain and the app.* subdomain as subjects. For example, if you host your installation at server.example.com, your certificate must cover app.server.example.com and server.example.com.

Google Cloud DNS

If you host your DNS on Google Cloud, you need the certbot-dns-google plugin installed. You can install the plugin with the following command:
```
pip3 install certbot-dns-google
```

Then, the following commands will provision a certification for your installation:

certbot certonly --dns-google --dns-google-credentials <PATH_TO_CREDENTIALS> -d "<CIRCLECI_SERVER_DOMAIN>" -d "app.<CIRCLECI_SERVER_DOMAIN>"

You will need these certificates later, and they can be retrieved locally with the following commands:

ls -l /etc/letsencrypt/live/<CIRCLECI_SERVER_DOMAIN>

cat /etc/letsencrypt/live/<CIRCLECI_SERVER_DOMAIN>/fullchain.pem

cat /etc/letsencrypt/live/<CIRCLECI_SERVER_DOMAIN>/privkey.pem

Encryption/signing keys

These keysets are used to encrypt and sign artifacts generated by CircleCI. You need these values to configure server.

Store these values securely. If they are lost, job history and artifacts will not be recoverable.

Artifact signing key

To generate an artifact signing key, run the following command:

docker run circleci/server-keysets:latest generate signing -a stdout

Encryption signing key

To generate an encryption signing key, run the following command:

docker run circleci/server-keysets:latest generate encryption -a stdout

Object storage and permissions

Server 3.x hosts build artifacts, test results, and other state object storage. We support the following:

While any S3 compatible object storage may work, we test and support AWS S3 and MinIO.

Please choose the option that best suits your needs. A Storage Bucket Name is required, in addition to the fields listed below, depending on whether you are using AWS or GCP. Before proceeding, ensure the bucket name you provide exists in your chosen object storage provider.

If you are installing behind a proxy, object storage should be behind this proxy also. Otherwise proxy details will need to be supplied at the job level within every project .circleci/config.yml to allow artifacts, test results, cache save and restore, and workspaces to work. For more information see the Configuring a Proxy guide.

Create an S3 storage bucket

You will need the following details when you configure CircleCI server.

Storage Bucket Name - The bucket name to be used for server.
Access Key ID - Access Key ID for S3 bucket access.
Secret Key - Secret Key for S3 bucket access.
AWS S3 Region - AWS region of bucket, if your provider is AWS. You will either have an AWS region or S3 Endpoint, depending on your specific setup.
S3 Endpoint - API endpoint of S3 storage provider, when your storage provider is not Amazon S3.

Steps to create your S3 bucket:

Create AWS S3 Bucket

aws s3api create-bucket \
    --bucket <YOUR_BUCKET_NAME> \
    --region <YOUR_REGION> \
    --create-bucket-configuration LocationConstraint=<YOUR_REGION>

us-east-1 does not support a LocationConstraint. If your region is us-east-1, omit the bucket configuration

Create an IAM user for CircleCI server

aws iam create-user --user-name circleci-server

Create a policy document policy.json

If using IAM Roles for Service Accounts (IRSA) for authentication, use the following content

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutAnalyticsConfiguration",
        "s3:GetObjectVersionTagging",
        "s3:CreateBucket",
        "s3:GetObjectAcl",
        "s3:GetBucketObjectLockConfiguration",
        "s3:DeleteBucketWebsite",
        "s3:PutLifecycleConfiguration",
        "s3:GetObjectVersionAcl",
        "s3:PutObjectTagging",
        "s3:DeleteObject",
        "s3:DeleteObjectTagging",
        "s3:GetBucketPolicyStatus",
        "s3:GetObjectRetention",
        "s3:GetBucketWebsite",
        "s3:GetJobTagging",
        "s3:DeleteObjectVersionTagging",
        "s3:PutObjectLegalHold",
        "s3:GetObjectLegalHold",
        "s3:GetBucketNotification",
        "s3:PutBucketCORS",
        "s3:GetReplicationConfiguration",
        "s3:ListMultipartUploadParts",
        "s3:PutObject",
        "s3:GetObject",
        "s3:PutBucketNotification",
        "s3:DescribeJob",
        "s3:PutBucketLogging",
        "s3:GetAnalyticsConfiguration",
        "s3:PutBucketObjectLockConfiguration",
        "s3:GetObjectVersionForReplication",
        "s3:GetLifecycleConfiguration",
        "s3:GetInventoryConfiguration",
        "s3:GetBucketTagging",
        "s3:PutAccelerateConfiguration",
        "s3:DeleteObjectVersion",
        "s3:GetBucketLogging",
        "s3:ListBucketVersions",
        "s3:ReplicateTags",
        "s3:RestoreObject",
        "s3:ListBucket",
        "s3:GetAccelerateConfiguration",
        "s3:GetBucketPolicy",
        "s3:PutEncryptionConfiguration",
        "s3:GetEncryptionConfiguration",
        "s3:GetObjectVersionTorrent",
        "s3:AbortMultipartUpload",
        "s3:PutBucketTagging",
        "s3:GetBucketRequestPayment",
        "s3:GetAccessPointPolicyStatus",
        "s3:GetObjectTagging",
        "s3:GetMetricsConfiguration",
        "s3:PutBucketVersioning",
        "s3:GetBucketPublicAccessBlock",
        "s3:ListBucketMultipartUploads",
        "s3:PutMetricsConfiguration",
        "s3:PutObjectVersionTagging",
        "s3:GetBucketVersioning",
        "s3:GetBucketAcl",
        "s3:PutInventoryConfiguration",
        "s3:GetObjectTorrent",
        "s3:PutBucketWebsite",
        "s3:PutBucketRequestPayment",
        "s3:PutObjectRetention",
        "s3:GetBucketCORS",
        "s3:GetBucketLocation",
        "s3:GetAccessPointPolicy",
        "s3:GetObjectVersion",
        "s3:GetAccessPoint",
        "s3:GetAccountPublicAccessBlock",
        "s3:ListAllMyBuckets",
        "s3:ListAccessPoints",
        "s3:ListJobs"
      ],
      "Resource": [
        "arn:aws:s3:::<YOUR_BUCKET_NAME>",
        "arn:aws:s3:::<YOUR_BUCKET_NAME>/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "iam:GetRole",
        "sts:AssumeRole"
      ],
      "Resource": "<YOUR_OBJECT_STORAGE_ROLE>"
    }
  ]
}

Otherwise, if using IAM keys for authentication, use the following content

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutAnalyticsConfiguration",
        "s3:GetObjectVersionTagging",
        "s3:CreateBucket",
        "s3:GetObjectAcl",
        "s3:GetBucketObjectLockConfiguration",
        "s3:DeleteBucketWebsite",
        "s3:PutLifecycleConfiguration",
        "s3:GetObjectVersionAcl",
        "s3:PutObjectTagging",
        "s3:DeleteObject",
        "s3:DeleteObjectTagging",
        "s3:GetBucketPolicyStatus",
        "s3:GetObjectRetention",
        "s3:GetBucketWebsite",
        "s3:GetJobTagging",
        "s3:DeleteObjectVersionTagging",
        "s3:PutObjectLegalHold",
        "s3:GetObjectLegalHold",
        "s3:GetBucketNotification",
        "s3:PutBucketCORS",
        "s3:GetReplicationConfiguration",
        "s3:ListMultipartUploadParts",
        "s3:PutObject",
        "s3:GetObject",
        "s3:PutBucketNotification",
        "s3:DescribeJob",
        "s3:PutBucketLogging",
        "s3:GetAnalyticsConfiguration",
        "s3:PutBucketObjectLockConfiguration",
        "s3:GetObjectVersionForReplication",
        "s3:GetLifecycleConfiguration",
        "s3:GetInventoryConfiguration",
        "s3:GetBucketTagging",
        "s3:PutAccelerateConfiguration",
        "s3:DeleteObjectVersion",
        "s3:GetBucketLogging",
        "s3:ListBucketVersions",
        "s3:ReplicateTags",
        "s3:RestoreObject",
        "s3:ListBucket",
        "s3:GetAccelerateConfiguration",
        "s3:GetBucketPolicy",
        "s3:PutEncryptionConfiguration",
        "s3:GetEncryptionConfiguration",
        "s3:GetObjectVersionTorrent",
        "s3:AbortMultipartUpload",
        "s3:PutBucketTagging",
        "s3:GetBucketRequestPayment",
        "s3:GetAccessPointPolicyStatus",
        "s3:GetObjectTagging",
        "s3:GetMetricsConfiguration",
        "s3:PutBucketVersioning",
        "s3:GetBucketPublicAccessBlock",
        "s3:ListBucketMultipartUploads",
        "s3:PutMetricsConfiguration",
        "s3:PutObjectVersionTagging",
        "s3:GetBucketVersioning",
        "s3:GetBucketAcl",
        "s3:PutInventoryConfiguration",
        "s3:GetObjectTorrent",
        "s3:PutBucketWebsite",
        "s3:PutBucketRequestPayment",
        "s3:PutObjectRetention",
        "s3:GetBucketCORS",
        "s3:GetBucketLocation",
        "s3:GetAccessPointPolicy",
        "s3:GetObjectVersion",
        "s3:GetAccessPoint",
        "s3:GetAccountPublicAccessBlock",
        "s3:ListAllMyBuckets",
        "s3:ListAccessPoints",
        "s3:ListJobs"
      ],
      "Resource": [
        "arn:aws:s3:::<YOUR_BUCKET_NAME>",
        "arn:aws:s3:::<YOUR_BUCKET_NAME>/*"
      ]
    }
  ]
}

Attach policy to user

aws iam put-user-policy \
  --user-name circleci-server \
  --policy-name circleci-server \
  --policy-document file://policy.json

Create Access Key for user circleci-server

You will need this when you configure your server installation later.

aws iam create-access-key --user-name circleci-server

The result should look like this:

{
  "AccessKey": {
        "UserName": "circleci-server",
        "Status": "Active",
        "CreateDate": "2017-07-31T22:24:41.576Z",
        "SecretAccessKey": <AWS_SECRET_ACCESS_KEY>,
        "AccessKeyId": <AWS_ACCESS_KEY_ID>
  }
}

Create a Google Cloud storage bucket

You will need the following details when you configure CircleCI server.

Storage Bucket Name - The bucket used for server.
You can choose one of the following:
- Service Account JSON - A JSON format key of the Service Account to use for bucket access.
- Service Account Email - Service Account Email id if using Google Workload Identity.

A dedicated service account is recommended. Add to it the Storage Object Admin role, with a condition on the resource name limiting access to only the bucket specified above. For example, enter the following into the Google’s Condition Editor in the IAM console:

Use startsWith and prefix the bucket name with projects/_/buckets/.

resource.name.startsWith("projects/_/buckets/<YOUR_BUCKET_NAME>")

Create a GCP bucket

If your server installation runs within a GKE cluster, ensure that your current IAM user is a cluster admin for this cluster, as RBAC (role-based access control) objects need to be created. More information can be found in the GKE documentation.
```
gsutil mb gs://circleci-server-bucket
```

Create a Service Account

gcloud iam service-accounts create circleci-server --display-name "circleci-server service account"

You will need the email for the service account in the next step. Run the following command to find it:

gcloud iam service-accounts list \
  --filter="displayName:circleci-server account" \
  --format 'value(email)'

Grant Permissions to Service Account

gcloud iam roles create circleci_server \
    --project <PROJECT_ID> \
    --title "CircleCI Server"

gcloud projects add-iam-policy-binding <PROJECT_ID> \
    --member serviceAccount:<SERVICE_ACCOUNT_EMAIL> \
    --role projects/<PROJECT_ID>/roles/circleci_server

gsutil iam ch serviceAccount:<SERVICE_ACCOUNT_EMAIL>:objectAdmin gs://circleci-server-bucket

JSON Key File

This step is NOT required if using Workload Identities.

After running the following command, you should have a file named circleci-server-keyfile in your local working directory. You will need this when you configure your server installation.
```
gcloud iam service-accounts keys create circleci-server-keyfile \
    --iam-account <SERVICE_ACCOUNT_EMAIL>
```

Enable workload Identity

This step is required only if you are using Workload Identities for GKE. Steps to enable Workload Identities are here

gcloud iam service-accounts add-iam-policy-binding <YOUR_SERVICE_ACCOUNT_EMAIL> \
    --role roles/iam.workloadIdentityUser \
    --member "serviceAccount:<GCP_PROJECT_ID>.svc.id.goog[circleci-server/object-storage]"

gcloud projects add-iam-policy-binding <GCP_PROJECT_ID> \
    --member serviceAccount:<YOUR_SERVICE_ACCOUNT_EMAIL> \
    --role roles/iam.serviceAccountTokenCreator \
    --condition=None

If you are switching from static JSON credentials to Workload Identity, you should delete the keys from GCP as well as from CircleCI KOTS Admin Console.

What to read next

Server 3.x Phase 2: Core services installation

Help make this document better

This guide, as well as the rest of our docs, are open source and available on GitHub. We welcome your contributions.

Suggest an edit to this page (please read the contributing guide first).
To report a problem in the documentation, or to submit feedback and comments, please open an issue on GitHub.
CircleCI is always seeking ways to improve your experience with our platform. If you would like to share feedback, please join our research community.

Need support?

Our support engineers are available to help with service issues, billing, or account related questions, and can help troubleshoot build configurations. Contact our support engineers by opening a ticket.

You can also visit our support site to find support articles, community forums, and training resources.

CircleCI Documentation by CircleCI is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.