Phase 1 AWS - Prerequisites
1 month ago
CircleCI server 4.8.0 is installed as a Helm chart. The installation process is broken down into four phases. There is a validation step at the end of each phase, allowing you to confirm success before moving to the next phase. Depending on your requirements, phases 3 and 4 may include multiple steps. This installation guide assumes you have already read the CircleCI server v4.8 overview.
In the following sections, replace any sections indicated by < > with your details.
|
1. Install required software
Download and install the following software before continuing:
| Tool | Version | Used for | Notes |
|---|---|---|---|
1.19 or greater |
Kubernetes CLI |
||
3.9.2 or greater |
Kubernetes Package Management |
||
3.5.0 or greater |
Helping with |
Optional, but may help with troubleshooting between releases |
|
Latest |
Managing AWS resources |
AWS installs only |
|
0.15.4 or greater |
Infrastructure Management |
Required for installations within GCP or AWS |
|
Install and configure MinIO CLI for your storage provider if you choose not to use AWS or GCP object storage. |
TBC |
TBC |
Required for installations outside AWS and GCP, for example, local installation. |
2. Create a VPC
| Installing server locally? you can skip this step, but you should ensure you follow the resourcing recommendation of using /18 CIDR blocks to ensure you have adequate capacity for both your Kubernetes Cluster and the provisioning of Nomad clients to run your jobs. |
If you are installing server in the cloud, either AWS or GCP, you will need to create a new virtual private cloud (VPC).
Refer to the AWS guide to creating a VPC.
VPC and cluster sizing recommendations
While there are no strict requirements for VPC (Virtual Private Cloud) setup or disk size, the following practices are recommended for optimal performance and stability in production environments.
-
VPC Selection
We recommend you create a new VPC specifically for CircleCI server and its components, rather than using an existing VPC.
-
Subnet Provisioning
For high availability, and to avoid potential outages, you should provision subnets using /18 CIDR blocks across multiple Availability Zones.
2. Create a Kubernetes cluster
CircleCI server installs into an existing Kubernetes cluster. If you have not already created a cluster, you should do so next. There are instructions for creating a cluster below, or if you are installing locally, first consider the sizing, version, and permissions requirements.
Cluster requirements
EKS
You can learn more about creating an Amazon EKS cluster in the EKS docs. We recommend using eksctl to create your cluster, which creates a VPC and selects the proper security groups for you.
-
Install
eksctl. -
Create your cluster by running the following (Cloud formation with
eksctland EKS can take more than 20 minutes to complete):$ eksctl create cluster --name=circleci-server --nodes 4 --node-type m5.2xlarge --with-oidc --region "<your-region>"--with-oidcas in the example above is used to add an IAM OIDC provider to your cluster. This provider is necessary for adding the EBS CSI driver and enabling IRSA roles on your cluster. See below for more information.eksctlmay be further extended with other flags to customize your EKS cluster following their documentation -
Once the cluster has been created, you can use the following command to configure
kubectlaccess:$ eksctl utils write-kubeconfig --cluster circleci-server -
As of EKS 1.24, EKS will no longer have the EBS CSI Driver installed by default. This means that EKS cannot manage the EBS volumes for the persistent volumes of your cluster. CircleCI uses persistent volumes for services such as PostgreSQL, MongoDB, Redis, RabbitMQ and Vault. You may manually provision persistence volumes or you may install the EBS CSI Driver on your cluster following the AWS documentation.
You may see the following error AWS STS Access - cannot get role ARN for current session: InvalidClientTokenID. This means your AWS credentials are invalid, or your IAM user does not have permission to create an EKS cluster. Proper IAM permissions are necessary to use eksctl. See the AWS documentation on IAM permissions.
|
3. Create a new GitHub OAuth app
| If GitHub Enterprise and CircleCI server are not on the same domain, then images and icons from GHE will fail to load in the CircleCI web app. |
Registering and setting up a new GitHub OAuth app for CircleCI server allows for authorization control to your server installation using GitHub OAuth and for updates to GitHub projects/repos using build status information. The following steps apply for both GitHub.com and GitHub Enterprise.
-
In your browser, navigate to and select New OAuth App.
Figure 1. New GitHub OAuth App -
Complete the following fields, based on your planned installation:
-
Homepage URL: The URL of your planned CircleCI installation.
-
Authorization callback URL: The authorization callback URL is the URL of your planned CircleCI installation followed by
/auth/github
-
-
Once completed, you will be shown the Client ID. Select Generate a new Client Secret to generate a Client Secret for your new OAuth App. You will need these values when you configure CircleCI server.
Figure 2. Client ID and Secret
GitHub Enterprise
If using GitHub Enterprise, you also need a personal access token and the domain name of your GitHub Enterprise instance.
Create the defaultToken by navigating to User Settings > Developer Settings > Personal access tokens. The default token requires no scopes. You will need this value when you configure CircleCI server.
4. Static IP addresses
It is recommended to provision a static IP address to assign to the load balancer created by the cluster. While this is not necessary, it does eliminate the need to update DNS records if the service-created load balancer is reprovisioned.
AWS: Reserve an elastic IP address
To reserve an elastic IP address in AWS, run the following AWS CLI commands in your desired environment.
This command needs to be run to generate an address for every subnet the load balancer deploys into - default 3.
# Run x times per x subnets (default 3)
aws ec2 allocate-address
# {
# "PublicIp": "10.0.0.1,
# "AllocationId": "eipalloc-12345",
# "PublicIpv4Pool": "amazon",
# "NetworkBorderGroup": "us-east-1",
# "Domain": "vpc"
#}Make note of each of the returned AllocationId values from the CLI for use in the values.yaml file.
5. Frontend TLS certificates
By default, CircleCI server creates self-signed certificates to get you started. In production, you should supply a certificate from a trusted certificate authority. The Let’s Encrypt certificate authority, for example, can issue a free certificate using their Certbot tool. The sections below cover using Google Cloud DNS and AWS Route 53.
It is important that your certificate contains both your domain and the app.* subdomain as subjects. For example, if you host your installation at server.example.com, your certificate must cover app.server.example.com and server.example.com.
|
Once you have created your certificates using one of the methods described below, you can use the following commands to retrieve the certificates later when you need them during this installation:
$ ls -l /etc/letsencrypt/live/<CIRCLECI_SERVER_DOMAIN>$ cat /etc/letsencrypt/live/<CIRCLECI_SERVER_DOMAIN>/fullchain.pem$ cat /etc/letsencrypt/live/<CIRCLECI_SERVER_DOMAIN>/privkey.pemAWS Route 53
-
If you are using AWS Route 53 for DNS, you need the
certbot-route53plugin installed. You can install the plugin with the following command:$ python3 -m pip install certbot-dns-route53 -
Then execute this example to create a private key and certificate (including intermediate certificates) locally in
/etc/letsencrypt/live/<CIRCLECI_SERVER_DOMAIN>:$ certbot certonly --dns-route53 -d "<CIRCLECI_SERVER_DOMAIN>" -d "app.<CIRCLECI_SERVER_DOMAIN>"
AWS Certificate Manager
Instead of provisioning your own TLS certificates, if you are setting up CircleCI server in an AWS environment, you can have AWS provision TLS certificates using Certificate Manager.
$ aws acm request-certificate \
--domain-name <CIRCLECI_SERVER_DOMAIN> \
--subject-alternative-names app.<CIRCLECI_SERVER_DOMAIN> \
--validation-method DNS \
--idempotency-token circleAfter running this command, navigate to the Certificate Manager AWS console and follow the wizard to provision the required DNS validation records with Route53. Take note of the ARN of the certificate once it is issued.
Upstream TLS termination
You may have a requirement to terminate TLS for CircleCI server outside the application. This is an alternate method to using ACM or supplying the certificate chain during Helm deployment. An example would be a proxy running in front of the CircleCI installation providing TLS termination for your domain name. In this case the CircleCI application acts as the backend for your load balancer or proxy.
CircleCI server listens on the following port numbers, which need to be configured depending how you are routing the traffic:
-
Frontend / API Gateway [TCP 80, 443]
-
Nomad server [TCP 4647]
Depending on your requirements you may choose to terminate TLS for only the frontend/api-gateway or provide TLS for services listening on all the ports.
6. Encryption/signing keys
The keysets generated in this section are used to encrypt and sign artifacts generated by CircleCI. You will need these values to configure server.
| Store these values securely. If they are lost, job history and artifacts will not be recoverable. |
7. Object storage and permissions
CircleCI server v4.8 hosts build artifacts, test results, and other state object storage. The following storage options are supported:
While any S3 compatible object storage may work, we test and support AWS S3 and MinIO. Follow the instructions below to create a bucket and access method for AWS S3 or GCS.
If you are installing locally rather than in AWS or GCP, follow the MinIO instructions in the Air-gapped installation docs.
If you are installing behind a proxy, object storage should be behind this proxy also. Otherwise, proxy details will need to be supplied at the job level within every project .circleci/config.yml to allow artifacts, test results, cache save and restore, and workspaces to work. For more information see the Installing server behind a proxy guide.
|
AWS S3 storage
a. Create AWS S3 bucket
$ aws s3api create-bucket \
--bucket <YOUR_BUCKET_NAME> \
--region <YOUR_REGION> \
--create-bucket-configuration LocationConstraint=<YOUR_REGION>b. Enable bucket versioning
To use the Docker layer caching (DLC) feature in CircleCI, bucket versioning needs to be enabled. Run the following command to enable bucket versioning on the bucket created in the previous step:
$ aws s3api put-bucket-versioning \
--bucket <YOUR_BUCKET_NAME> \
--region <YOUR_REGION> \
--versioning-configuration Status=Enabledc. Set up authentication
Authenticate CircleCI with S3 in one of two ways:
-
IAM Roles for Service Accounts (IRSA) - recommended
-
IAM access keys
-
IRSA
-
IAM access keys
Option 1: IRSA
The following is a summary of AWS’s Documentation on IRSA that is sufficient for installing CircleCI.
-
Create an IAM OIDC Identity Provider for your EKS Cluster.
$ eksctl utils associate-iam-oidc-provider --cluster <CLUSTER_NAME> --approve -
Get the OIDC provider ARN. You will need this in later steps.
$ aws iam list-open-id-connect-providers | grep $(aws eks describe-cluster --name <CLUSTER_NAME> --query "cluster.identity.oidc.issuer" --output text | awk -F'/' '{print $NF}') -
Get your OIDC provider URL. You will need this in later steps.
$ aws eks describe-cluster --name <CLUSTER_NAME> --query "cluster.identity.oidc.issuer" --output text | sed -e "s/^https:\/\///" -
Create the role using the command and trust policy template below. You will need the Role ARN and name in later steps.
$ aws iam create-role --role-name circleci-s3 --assume-role-policy-document file://<TRUST_POLICY_FILE>{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "<OIDC_PROVIDER_ARN>" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "<OIDC_PROVIDER_URL>:sub": "system:serviceaccount:<K8S_NAMESPACE>:object-storage" } } } ] }If you wish to store artifacts which are larger than 5GB, you will need to disable presigned mode. To do this you will need your IRSA role to assume itself. Replace your trust policy above with the contents below. { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "<OIDC_PROVIDER_ARN>" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "<OIDC_PROVIDER_URL>:sub": "system:serviceaccount:<K8S_NAMESPACE>:object-storage" } } }, { "Effect": "Allow", "Principal": { "AWS": "<ROLE_ARN>" }, "Action": "sts:AssumeRole" } ] } -
Create the policy using the command and template below. Fill in the bucket name and the role ARN.
$ aws iam create-policy --policy-name circleci-s3 --policy-document file://<POLICY_FILE>{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:PutAnalyticsConfiguration", "s3:GetObjectVersionTagging", "s3:CreateBucket", "s3:GetObjectAcl", "s3:GetBucketObjectLockConfiguration", "s3:DeleteBucketWebsite", "s3:PutLifecycleConfiguration", "s3:GetObjectVersionAcl", "s3:PutObjectTagging", "s3:DeleteObject", "s3:DeleteObjectTagging", "s3:GetBucketPolicyStatus", "s3:GetObjectRetention", "s3:GetBucketWebsite", "s3:GetJobTagging", "s3:DeleteObjectVersionTagging", "s3:PutObjectLegalHold", "s3:GetObjectLegalHold", "s3:GetBucketNotification", "s3:PutBucketCORS", "s3:GetReplicationConfiguration", "s3:ListMultipartUploadParts", "s3:PutObject", "s3:GetObject", "s3:PutBucketNotification", "s3:DescribeJob", "s3:PutBucketLogging", "s3:GetAnalyticsConfiguration", "s3:PutBucketObjectLockConfiguration", "s3:GetObjectVersionForReplication", "s3:GetLifecycleConfiguration", "s3:GetInventoryConfiguration", "s3:GetBucketTagging", "s3:PutAccelerateConfiguration", "s3:DeleteObjectVersion", "s3:GetBucketLogging", "s3:ListBucketVersions", "s3:ReplicateTags", "s3:RestoreObject", "s3:ListBucket", "s3:GetAccelerateConfiguration", "s3:GetBucketPolicy", "s3:PutEncryptionConfiguration", "s3:GetEncryptionConfiguration", "s3:GetObjectVersionTorrent", "s3:AbortMultipartUpload", "s3:PutBucketTagging", "s3:GetBucketRequestPayment", "s3:GetAccessPointPolicyStatus", "s3:GetObjectTagging", "s3:GetMetricsConfiguration", "s3:PutBucketVersioning", "s3:GetBucketPublicAccessBlock", "s3:ListBucketMultipartUploads", "s3:PutMetricsConfiguration", "s3:PutObjectVersionTagging", "s3:GetBucketVersioning", "s3:GetBucketAcl", "s3:PutInventoryConfiguration", "s3:GetObjectTorrent", "s3:PutBucketWebsite", "s3:PutBucketRequestPayment", "s3:PutObjectRetention", "s3:GetBucketCORS", "s3:GetBucketLocation", "s3:GetAccessPointPolicy", "s3:GetObjectVersion", "s3:GetAccessPoint", "s3:GetAccountPublicAccessBlock", "s3:ListAllMyBuckets", "s3:ListAccessPoints", "s3:ListJobs" ], "Resource": [ "arn:aws:s3:::<YOUR_BUCKET_NAME>", "arn:aws:s3:::<YOUR_BUCKET_NAME>/*" ] }, { "Effect": "Allow", "Action": [ "iam:GetRole", "sts:AssumeRole" ], "Resource": "<OBJECT_STORAGE_ROLE_ARN>" } ] } -
Attach the policy to the role:
$ aws iam attach-role-policy --role-name <OBJECT_STORAGE_ROLE_NAME> --policy-arn=<STORAGE_POLICY_ARN>
Option 2: IAM access keys
| If you wish to store artifacts which are larger than 5GB, you will need to disable presigned mode which requires an AWS role. Disabling presigned mode will enable the use of muli-part uploads to S3 which can support larger files and potentially faster transfers. We recommend you follow the instructions for creating an IRSA role in this case. |
-
Create an IAM user for CircleCI server.
$ aws iam create-user --user-name circleci-server -
Create a policy document
policy.json.{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:PutAnalyticsConfiguration", "s3:GetObjectVersionTagging", "s3:CreateBucket", "s3:GetObjectAcl", "s3:GetBucketObjectLockConfiguration", "s3:DeleteBucketWebsite", "s3:PutLifecycleConfiguration", "s3:GetObjectVersionAcl", "s3:PutObjectTagging", "s3:DeleteObject", "s3:DeleteObjectTagging", "s3:GetBucketPolicyStatus", "s3:GetObjectRetention", "s3:GetBucketWebsite", "s3:GetJobTagging", "s3:DeleteObjectVersionTagging", "s3:PutObjectLegalHold", "s3:GetObjectLegalHold", "s3:GetBucketNotification", "s3:PutBucketCORS", "s3:GetReplicationConfiguration", "s3:ListMultipartUploadParts", "s3:PutObject", "s3:GetObject", "s3:PutBucketNotification", "s3:DescribeJob", "s3:PutBucketLogging", "s3:GetAnalyticsConfiguration", "s3:PutBucketObjectLockConfiguration", "s3:GetObjectVersionForReplication", "s3:GetLifecycleConfiguration", "s3:GetInventoryConfiguration", "s3:GetBucketTagging", "s3:PutAccelerateConfiguration", "s3:DeleteObjectVersion", "s3:GetBucketLogging", "s3:ListBucketVersions", "s3:ReplicateTags", "s3:RestoreObject", "s3:ListBucket", "s3:GetAccelerateConfiguration", "s3:GetBucketPolicy", "s3:PutEncryptionConfiguration", "s3:GetEncryptionConfiguration", "s3:GetObjectVersionTorrent", "s3:AbortMultipartUpload", "s3:PutBucketTagging", "s3:GetBucketRequestPayment", "s3:GetAccessPointPolicyStatus", "s3:GetObjectTagging", "s3:GetMetricsConfiguration", "s3:PutBucketVersioning", "s3:GetBucketPublicAccessBlock", "s3:ListBucketMultipartUploads", "s3:PutMetricsConfiguration", "s3:PutObjectVersionTagging", "s3:GetBucketVersioning", "s3:GetBucketAcl", "s3:PutInventoryConfiguration", "s3:GetObjectTorrent", "s3:PutBucketWebsite", "s3:PutBucketRequestPayment", "s3:PutObjectRetention", "s3:GetBucketCORS", "s3:GetBucketLocation", "s3:GetAccessPointPolicy", "s3:GetObjectVersion", "s3:GetAccessPoint", "s3:GetAccountPublicAccessBlock", "s3:ListAllMyBuckets", "s3:ListAccessPoints", "s3:ListJobs" ], "Resource": [ "arn:aws:s3:::<YOUR_BUCKET_NAME>", "arn:aws:s3:::<YOUR_BUCKET_NAME>/*" ] } ] } -
Attach policy to user.
$ aws iam put-user-policy \ --user-name circleci-server \ --policy-name circleci-server \ --policy-document file://policy.json -
Create Access Key for user
circleci-server.You will need this when you configure your server installation later. $ aws iam create-access-key --user-name circleci-serverThe result should look like this:
{ "AccessKey": { "UserName": "circleci-server", "Status": "Active", "CreateDate": "2017-07-31T22:24:41.576Z", "SecretAccessKey": <AWS_SECRET_ACCESS_KEY>, "AccessKeyId": <AWS_ACCESS_KEY_ID> } }