Start Building for Free
CircleCI.comAcademyBlogCommunitySupport

CircleCI Server v3.x Installation Phase 1

2 months ago8 min read
Server v3.x
Server Admin
On This Page

Phase 1: Prerequisites

CircleCI server v3.x is installed in 4 phases. There is a validation step at the end of each phase, allowing you to confirm success before moving to the next phase. Depending on your requirements, phases 3 and 4 may include multiple steps. This installation guide assumes you have already read the CircleCI Server v3.x Overview.

Flow chart showing the installation flow for server 3.x with phase 1 highlighted
Figure 1. Installation Experience Flow Chart Phase 1

Install required software

Download and install the following software before continuing:

ToolVersionUsed forNotes

Terraform

0.15.4 or greater

Infrastructure Management

kubectl

1.19 or greater

Kubernetes CLI

Helm

3.9.2 or greater

Kubernetes Package Management

KOTS: Mac or Linux.

1.65.0 *

Replicated Kubernetes Application Management. KOTS is a kubectl plugin.

Once you have extracted kots from the tar.gz (tar zxvf kots_linux_amd64.tar.gz), run sudo mv kots /usr/local/bin/kubectl-kots to install it. Mac users will need to grant a security exception.

Velero CLI

Latest

Backup and restore capability

See Velero’s supported providers documentation for further information.

* Please take note of the supported KOTS versions for your Kubernetes cluster. KOTS version compatibility

AWS required software

GCP required software

  • gcloud and gsutil. You can install and set-up these tools up by installing Google Cloud SDK. For further information refer to the Google Cloud SDK docs.

S3 compatible storage required software

  • Install and configure MinIO CLI for your storage provider.

Create a Kubernetes cluster

CircleCI server installs into an existing Kubernetes cluster. The application uses a large number of resources. Depending on your usage, your Kubernetes cluster should meet the following requirements:

Number of daily active CircleCI usersMinimum NodesTotal CPUTotal RAMNIC speed

< 500

3

12 cores

32 GB

1 Gbps

500+

3

48 cores

240 GB

10 Gbps

Supported Kubernetes versions:

CircleCI VersionKubernetes Version

3.0.0 - 3.2.1

< 1.21

3.2.2 - 3.3.0

1.16 - 1.21

3.4.0 - 3.4.6

1.16 - 1.23

Creating a Kubernetes cluster is your responsibility. Please note:

  • Your cluster must have outbound access to pull Docker containers and verify your license. If you do not want to provide open outbound access, see our list of ports that need access.

  • You must have appropriate permissions to list, create, edit, and delete pods in your cluster. Run this command to verify your permissions:

    kubectl auth can-i <list|create|edit|delete> pods
  • There are no requirements regarding VPC setup or disk size for your cluster. It is recommended that you set up a new VPC rather than use an existing one.

EKS

You can learn more about creating an Amazon EKS cluster here. We recommend using eksctl to create your cluster, which creates a VPC and selects the proper security groups for you.

  1. Install and configure the AWS CLI for your AWS account.

  2. Install eksctl.

  3. Create your cluster by running the following (Cloud formation with eksctl and EKS can take more than 20 minutes to complete):

    eksctl create cluster --name=circleci-server --nodes 4 --node-type m5.xlarge
  4. Once the cluster has been created, you can use the following command to configure kubectl access:

    eksctl utils write-kubeconfig --cluster circleci-server

GKE

You can learn more about creating a GKE cluster here.

  1. Install and configure the GCP CLI for your GCP account. This includes creating a Google Project, which will be required to create a cluster within your project.

  2. Create your cluster by running the following command:

    gcloud container clusters create circleci-server --project <YOUR_GOOGLE_CLOUD_PROJECT_ID> --region europe-west1 --num-nodes 3 --machine-type n1-standard-4
  3. Configure kubectl with your your gcloud credentials:

    gcloud container clusters get-credentials circleci-server --region europe-west1
  4. Verify your cluster:

    kubectl cluster-info
  5. Create a service account for this cluster:

    gcloud iam service-accounts create <YOUR_SERVICE_ACCOUNT_ID> --description="<YOUR_SERVICE_ACCOUNT_DESCRIPTION>"  --display-name="<YOUR_SERVICE_ACCOUNT_DISPLAY_NAME>"
  6. Get the credentials for the service account:

    gcloud iam service-accounts keys create <PATH_TO_STORE_CREDENTIALS> --iam-account <SERVICE_ACCOUNT_ID>@<YOUR_GOOGLE_CLOUD_PROJECT_ID>.iam.gserviceaccount.com
Enable Workload Identities in GKE (optional)

Workload Identities for GKE allow workloads/pods in your GKE cluster to impersonate IAM service accounts to access Google Cloud services without using static service account credentials. In order to use Workload Identities you must enable them on your GKE cluster.

  1. Enable Workload Identity on existing cluster

      gcloud container clusters update "<CLUSTER_NAME>" \
        --region="<REGION>" \
        --workload-pool="<PROJECT_ID>.svc.id.goog"
  2. Get node pools of existing GKE cluster

      gcloud container node-pools list --cluster "<CLUSTER_NAME>" --region "<REGION>"
  3. Update existing node pools

      gcloud container node-pools update "<NODEPOOL_NAME>" \
        --cluster="<CLUSTER_NAME>" \
        --workload-metadata="GKE_METADATA" \
        --region="<REGION>"

You must repeat Step 3 for all the existing node pools. Follow these links for steps to enable Workload Identity for your Kubernetes service accounts: Nomad Autoscaler, VM and Object-Storage

Create a new GitHub OAuth app

Registering and setting up a new GitHub OAuth app for CircleCI server allows for authorization control to your server installation using GitHub OAuth and for updates to GitHub projects/repos using build status information.

  1. In your browser, navigate to your GitHub instance > Settings > Developer Settings > OAuth Apps and click the New OAuth App button.

    Screenshot showing setting up a new OAuth app
    Figure 2. New GitHub OAuth App
  2. Complete the following fields, based on your planned installation:

    • Homepage URL: The URL of your planned CircleCI installation.

    • Authorization callback URL: The authorization callback URL is the URL of your planned CircleCI installation followed by /auth/github

  3. Once completed, you will be shown the Client ID. Select Generate a new Client Secret to generate a Client Secret for your new OAuth App. You need these values when you configure CircleCI server.

    Screenshot showing GitHub Client ID
    Figure 3. Client ID and Secret

Frontend TLS certificates

By default, CircleCI server creates self-signed certificates to get you started. In production, you should supply a certificate from a trusted certificate authority. The Let’s Encrypt certificate authority, for example, can issue a free certificate using their certbot tool. The sections below cover using Google Cloud DNS and AWS Route 53.

AWS Route 53

  1. If you are using AWS Route 53 for DNS, you need the certbot-route53 plugin installed. You can install the plugin with the following command:

    pip3 install certbot-dns-route53
  2. Then execute this example to create a private key and certificate (including intermediate certificates) locally in /etc/letsencrypt/live/<CIRCLECI_SERVER_DOMAIN>:

    certbot certonly --dns-route53 -d "<CIRCLECI_SERVER_DOMAIN>" -d "app.<CIRCLECI_SERVER_DOMAIN>"

Google Cloud DNS

  1. If you host your DNS on Google Cloud, you need the certbot-dns-google plugin installed. You can install the plugin with the following command:

    pip3 install certbot-dns-google
  2. Then, the following commands will provision a certification for your installation:

    certbot certonly --dns-google --dns-google-credentials <PATH_TO_CREDENTIALS> -d "<CIRCLECI_SERVER_DOMAIN>" -d "app.<CIRCLECI_SERVER_DOMAIN>"

You will need these certificates later, and they can be retrieved locally with the following commands:

ls -l /etc/letsencrypt/live/<CIRCLECI_SERVER_DOMAIN>
cat /etc/letsencrypt/live/<CIRCLECI_SERVER_DOMAIN>/fullchain.pem
cat /etc/letsencrypt/live/<CIRCLECI_SERVER_DOMAIN>/privkey.pem

Encryption/signing keys

These keysets are used to encrypt and sign artifacts generated by CircleCI. You need these values to configure server.

Artifact signing key

To generate an artifact signing key, run the following command:

docker run circleci/server-keysets:latest generate signing -a stdout

Encryption signing key

To generate an encryption signing key, run the following command:

docker run circleci/server-keysets:latest generate encryption -a stdout

Object storage and permissions

Server 3.x hosts build artifacts, test results, and other state object storage. We support the following:

While any S3 compatible object storage may work, we test and support AWS S3 and MinIO.

Please choose the option that best suits your needs. A Storage Bucket Name is required, in addition to the fields listed below, depending on whether you are using AWS or GCP. Before proceeding, ensure the bucket name you provide exists in your chosen object storage provider.

Create an S3 storage bucket

You will need the following details when you configure CircleCI server.

  • Storage Bucket Name - The bucket name to be used for server.

  • Access Key ID - Access Key ID for S3 bucket access.

  • Secret Key - Secret Key for S3 bucket access.

  • AWS S3 Region - AWS region of bucket, if your provider is AWS. You will either have an AWS region or S3 Endpoint, depending on your specific setup.

  • S3 Endpoint - API endpoint of S3 storage provider, when your storage provider is not Amazon S3.

Steps to create your S3 bucket:

  1. Create AWS S3 Bucket

    aws s3api create-bucket \
        --bucket <YOUR_BUCKET_NAME> \
        --region <YOUR_REGION> \
        --create-bucket-configuration LocationConstraint=<YOUR_REGION>
  2. Create an IAM user for CircleCI server

    aws iam create-user --user-name circleci-server
  3. Create a policy document policy.json

    If using IAM Roles for Service Accounts (IRSA) for authentication, use the following content

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Action": [
            "s3:PutAnalyticsConfiguration",
            "s3:GetObjectVersionTagging",
            "s3:CreateBucket",
            "s3:GetObjectAcl",
            "s3:GetBucketObjectLockConfiguration",
            "s3:DeleteBucketWebsite",
            "s3:PutLifecycleConfiguration",
            "s3:GetObjectVersionAcl",
            "s3:PutObjectTagging",
            "s3:DeleteObject",
            "s3:DeleteObjectTagging",
            "s3:GetBucketPolicyStatus",
            "s3:GetObjectRetention",
            "s3:GetBucketWebsite",
            "s3:GetJobTagging",
            "s3:DeleteObjectVersionTagging",
            "s3:PutObjectLegalHold",
            "s3:GetObjectLegalHold",
            "s3:GetBucketNotification",
            "s3:PutBucketCORS",
            "s3:GetReplicationConfiguration",
            "s3:ListMultipartUploadParts",
            "s3:PutObject",
            "s3:GetObject",
            "s3:PutBucketNotification",
            "s3:DescribeJob",
            "s3:PutBucketLogging",
            "s3:GetAnalyticsConfiguration",
            "s3:PutBucketObjectLockConfiguration",
            "s3:GetObjectVersionForReplication",
            "s3:GetLifecycleConfiguration",
            "s3:GetInventoryConfiguration",
            "s3:GetBucketTagging",
            "s3:PutAccelerateConfiguration",
            "s3:DeleteObjectVersion",
            "s3:GetBucketLogging",
            "s3:ListBucketVersions",
            "s3:ReplicateTags",
            "s3:RestoreObject",
            "s3:ListBucket",
            "s3:GetAccelerateConfiguration",
            "s3:GetBucketPolicy",
            "s3:PutEncryptionConfiguration",
            "s3:GetEncryptionConfiguration",
            "s3:GetObjectVersionTorrent",
            "s3:AbortMultipartUpload",
            "s3:PutBucketTagging",
            "s3:GetBucketRequestPayment",
            "s3:GetAccessPointPolicyStatus",
            "s3:GetObjectTagging",
            "s3:GetMetricsConfiguration",
            "s3:PutBucketVersioning",
            "s3:GetBucketPublicAccessBlock",
            "s3:ListBucketMultipartUploads",
            "s3:PutMetricsConfiguration",
            "s3:PutObjectVersionTagging",
            "s3:GetBucketVersioning",
            "s3:GetBucketAcl",
            "s3:PutInventoryConfiguration",
            "s3:GetObjectTorrent",
            "s3:PutBucketWebsite",
            "s3:PutBucketRequestPayment",
            "s3:PutObjectRetention",
            "s3:GetBucketCORS",
            "s3:GetBucketLocation",
            "s3:GetAccessPointPolicy",
            "s3:GetObjectVersion",
            "s3:GetAccessPoint",
            "s3:GetAccountPublicAccessBlock",
            "s3:ListAllMyBuckets",
            "s3:ListAccessPoints",
            "s3:ListJobs"
          ],
          "Resource": [
            "arn:aws:s3:::<YOUR_BUCKET_NAME>",
            "arn:aws:s3:::<YOUR_BUCKET_NAME>/*"
          ]
        },
        {
          "Effect": "Allow",
          "Action": [
            "iam:GetRole",
            "sts:AssumeRole"
          ],
          "Resource": "<YOUR_OBJECT_STORAGE_ROLE>"
        }
      ]
    }

    Otherwise, if using IAM keys for authentication, use the following content

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Action": [
            "s3:PutAnalyticsConfiguration",
            "s3:GetObjectVersionTagging",
            "s3:CreateBucket",
            "s3:GetObjectAcl",
            "s3:GetBucketObjectLockConfiguration",
            "s3:DeleteBucketWebsite",
            "s3:PutLifecycleConfiguration",
            "s3:GetObjectVersionAcl",
            "s3:PutObjectTagging",
            "s3:DeleteObject",
            "s3:DeleteObjectTagging",
            "s3:GetBucketPolicyStatus",
            "s3:GetObjectRetention",
            "s3:GetBucketWebsite",
            "s3:GetJobTagging",
            "s3:DeleteObjectVersionTagging",
            "s3:PutObjectLegalHold",
            "s3:GetObjectLegalHold",
            "s3:GetBucketNotification",
            "s3:PutBucketCORS",
            "s3:GetReplicationConfiguration",
            "s3:ListMultipartUploadParts",
            "s3:PutObject",
            "s3:GetObject",
            "s3:PutBucketNotification",
            "s3:DescribeJob",
            "s3:PutBucketLogging",
            "s3:GetAnalyticsConfiguration",
            "s3:PutBucketObjectLockConfiguration",
            "s3:GetObjectVersionForReplication",
            "s3:GetLifecycleConfiguration",
            "s3:GetInventoryConfiguration",
            "s3:GetBucketTagging",
            "s3:PutAccelerateConfiguration",
            "s3:DeleteObjectVersion",
            "s3:GetBucketLogging",
            "s3:ListBucketVersions",
            "s3:ReplicateTags",
            "s3:RestoreObject",
            "s3:ListBucket",
            "s3:GetAccelerateConfiguration",
            "s3:GetBucketPolicy",
            "s3:PutEncryptionConfiguration",
            "s3:GetEncryptionConfiguration",
            "s3:GetObjectVersionTorrent",
            "s3:AbortMultipartUpload",
            "s3:PutBucketTagging",
            "s3:GetBucketRequestPayment",
            "s3:GetAccessPointPolicyStatus",
            "s3:GetObjectTagging",
            "s3:GetMetricsConfiguration",
            "s3:PutBucketVersioning",
            "s3:GetBucketPublicAccessBlock",
            "s3:ListBucketMultipartUploads",
            "s3:PutMetricsConfiguration",
            "s3:PutObjectVersionTagging",
            "s3:GetBucketVersioning",
            "s3:GetBucketAcl",
            "s3:PutInventoryConfiguration",
            "s3:GetObjectTorrent",
            "s3:PutBucketWebsite",
            "s3:PutBucketRequestPayment",
            "s3:PutObjectRetention",
            "s3:GetBucketCORS",
            "s3:GetBucketLocation",
            "s3:GetAccessPointPolicy",
            "s3:GetObjectVersion",
            "s3:GetAccessPoint",
            "s3:GetAccountPublicAccessBlock",
            "s3:ListAllMyBuckets",
            "s3:ListAccessPoints",
            "s3:ListJobs"
          ],
          "Resource": [
            "arn:aws:s3:::<YOUR_BUCKET_NAME>",
            "arn:aws:s3:::<YOUR_BUCKET_NAME>/*"
          ]
        }
      ]
    }
  4. Attach policy to user

    aws iam put-user-policy \
      --user-name circleci-server \
      --policy-name circleci-server \
      --policy-document file://policy.json
  5. Create Access Key for user circleci-server

    aws iam create-access-key --user-name circleci-server

    The result should look like this:

    {
      "AccessKey": {
            "UserName": "circleci-server",
            "Status": "Active",
            "CreateDate": "2017-07-31T22:24:41.576Z",
            "SecretAccessKey": <AWS_SECRET_ACCESS_KEY>,
            "AccessKeyId": <AWS_ACCESS_KEY_ID>
      }
    }

Create a Google Cloud storage bucket

You will need the following details when you configure CircleCI server.

  • Storage Bucket Name - The bucket used for server.

  • You can choose one of the following:

    • Service Account JSON - A JSON format key of the Service Account to use for bucket access.

    • Service Account Email - Service Account Email id if using Google Workload Identity.

A dedicated service account is recommended. Add to it the Storage Object Admin role, with a condition on the resource name limiting access to only the bucket specified above. For example, enter the following into the Google’s Condition Editor in the IAM console:

resource.name.startsWith("projects/_/buckets/<YOUR_BUCKET_NAME>")
  1. Create a GCP bucket

    If your server installation runs within a GKE cluster, ensure that your current IAM user is a cluster admin for this cluster, as RBAC (role-based access control) objects need to be created. More information can be found in the GKE documentation.

    gsutil mb gs://circleci-server-bucket
  2. Create a Service Account

    gcloud iam service-accounts create circleci-server --display-name "circleci-server service account"

    You will need the email for the service account in the next step. Run the following command to find it:

    gcloud iam service-accounts list \
      --filter="displayName:circleci-server account" \
      --format 'value(email)'
  3. Grant Permissions to Service Account

    gcloud iam roles create circleci_server \
        --project <PROJECT_ID> \
        --title "CircleCI Server"
    gcloud projects add-iam-policy-binding <PROJECT_ID> \
        --member serviceAccount:<SERVICE_ACCOUNT_EMAIL> \
        --role projects/<PROJECT_ID>/roles/circleci_server
    gsutil iam ch serviceAccount:<SERVICE_ACCOUNT_EMAIL>:objectAdmin gs://circleci-server-bucket
  4. JSON Key File

    This step is NOT required if using Workload Identities.

    After running the following command, you should have a file named circleci-server-keyfile in your local working directory. You will need this when you configure your server installation.

    gcloud iam service-accounts keys create circleci-server-keyfile \
        --iam-account <SERVICE_ACCOUNT_EMAIL>
  5. Enable workload Identity

    This step is required only if you are using Workload Identities for GKE. Steps to enable Workload Identities are here

    gcloud iam service-accounts add-iam-policy-binding <YOUR_SERVICE_ACCOUNT_EMAIL> \
        --role roles/iam.workloadIdentityUser \
        --member "serviceAccount:<GCP_PROJECT_ID>.svc.id.goog[circleci-server/object-storage]"
    gcloud projects add-iam-policy-binding <GCP_PROJECT_ID> \
        --member serviceAccount:<YOUR_SERVICE_ACCOUNT_EMAIL> \
        --role roles/iam.serviceAccountTokenCreator \
        --condition=None

Help make this document better

This guide, as well as the rest of our docs, are open source and available on GitHub. We welcome your contributions.

Need support?

Our support engineers are available to help with service issues, billing, or account related questions, and can help troubleshoot build configurations. Contact our support engineers by opening a ticket.

You can also visit our support site to find support articles, community forums, and training resources.