Phase 2 AWS - core services

Server v4.7

Server Admin

Before you begin with the CircleCI server v4.7 core services installation phase, ensure all prerequisites are met, as described in the following page:

Phase 1 - AWS Prerequisites

In the following sections, replace any sections indicated by < > with your details.

1. Create a namespace

Create a namespace to install the application into.

kubectl create ns <namespace>

Once you have created your namespace, we recommend setting your kubectl context too, with the following command: kubectl config set-context --current --namespace <namespace>

2. Pull images

Credentials to pull the images from CircleCI’s image registry will be provided to you as part of the onboarding process. A docker-registry Kubernetes Secret will be used to pull images from Azure Container Registry (ACR). You have two options, depending on whether your application has access to the public internet.

3. Create Helm values

Before installing CircleCI, it is recommended to create a new values.yaml file unique to your installation. The Installation Reference section contains some example values.yaml files that are a good place to start. The following describes the minimum required values to include in values.yaml. Additional customizations are available, see the provided values.yaml for all available options.

For sensitive data there are two options:

Add into the values.yaml file
Add them as Kubernetes Secrets directly

This flexibility allows you to manage Kubernetes Secrets using whichever process you prefer. Whichever option you choose, this sensitive information is stored as a Kubernetes Secret within CircleCI.

During the installation process, you may use the following command to generate a random alphanumeric value as required: cat /dev/urandom | LC_ALL=C tr -dc 'a-zA-Z0-9' | head -c <num-of-chars>. This command should work on any *nix based system.

a. API token

The application requires a Kubernetes Secret containing an API token. This API token is used to facilitate internal API communication to api-service. Use a random string and store it securely. CircleCI will not be able to recover this value if lost. There are two options depending on whether you want to create the Kubernetes Secret, or if you want CircleCI to create it for you.

The application requires a session cookie key Kubernetes Secret, which CircleCI uses to sign session cookies. The Secret must be exactly 16 characters long. Use a random string and store it securely. CircleCI will not be able to recover this value if lost. There are two options depending on whether you want to create the Kubernetes Secret, or if you want CircleCI to create it for you.

c. Encryption

The application requires a Kubernetes Secret containing signing and encryption keysets. These keysets are used to encrypt and sign artifacts generated by CircleCI. These keys were created during the prerequisites phase ( GCP prerequisites, AWS prerequisites). CircleCI will not be able to recover the values if lost. Depending on how you prefer to manage Kubernetes Secrets, there are two options.

d. PostgreSQL

i. Credentials

The application requires a Kubernetes Secret containing PostgreSQL credentials. This is true when using either the internal (default) or an externally hosted instance of PostgreSQL. CircleCI will not be able to recover the values if lost. Based on how you prefer to manage Kubernetes Secrets there are two options.

ii. TLS

PostgreSQL may be extended to use TLS encrypted traffic. When deployed internally, this option is disabled by default but may be enabled by adding the following to your PostgreSQL block of your values.yaml

postgresql:
  ...
  tls:
    enabled: true
    autoGenerated: true # Generate automatically self-signed TLS certificates

Certificate files may also be provided, rather than autogenerated. In this case, create a Secret containing the TLS certs and keys needed.

kubectl -n <namespace> create secret generic postgres-tls-secret --from-file=./cert.pem --from-file=./cert.key --from-file=./ca.pem

Then the PostgreSQL block in your values.yaml will contain the contents below.

postgresql:
  ...
  tls:
    enabled: true
    certificatesSecret: "postgres-tls-secret" # Name of an existing secret that contains the certificates
    certFilename: "cert.pem" # Certificate filename
    certKeyFilename: "cert.key" # Certificate key filename
    certCAFilename: "ca.pem" # CA Certificate filename

e. MongoDB credentials

The application requires a Kubernetes Secret containing MongoDB credentials. This is true when using either the internal (default) or an externally hosted instance of MongoDB. CircleCI will not be able to recover the values if lost. Based on how you prefer to manage Kubernetes Secrets there are two options.

f. RabbitMQ configurations and auth Secrets

The RabbitMQ installation requires two random alphanumeric strings. CircleCI will not be able to recover the values if lost. Based on how you prefer to manage Kubernetes Secrets there are two options.

g. Pusher Kubernetes Secret

The application requires a Kubernetes Secret for Pusher. CircleCI will not be able to recover the values if lost. Based on how you prefer to manage Kubernetes Secrets there are 2 options:

h. Global

All values in this section are children of global in your values.yaml.

CircleCI domain name (required)

Enter the domain name you specified when creating your Frontend TLS key and certificate ( AWS, GCP).

global:
  ...
  domainName: "<full-domain-name-of-your-install>"

License

A license will be provided by CircleCI, add it to values.yaml:

global:
  ...
  license: '<license>'

Registry

The registry to pull images from will have been provided to you, or you may have added the images to your own hosted registry. Add the registry to values.yaml:

global:
  ...
  container:
    registry: <registry-domain eg: cciserver.azurecr.io >
    org: <your-org-if-applicable>

i. Add Elastic IP addresses

If you provisioned an Elastic IP addresses (AWS) in the prerequisites, you can now add the values under the nginx block.

For AWS, under nginx annotations, add the service.beta.kubernetes.io/aws-load-balancer-eip-allocations annotation with each of the AllocationId values generated as a comma separated list. The number of `AllocationId`s must match the number of subnets the load balancer is deployed into (default 3).

nginx:
  ...
  annotations:
    ...
    service.beta.kubernetes.io/aws-load-balancer-eip-allocations: <eip-id-1>,<eip-id-2>,<eip-id-3>

j. TLS

For TLS, you have a few options:

k. GitHub integration

To configure GitHub with CircleCI, there are two options for providing credentials to the deployment. Steps for both GitHub and GitHub Enterprise (GHE) are given in the next two sections.

i. GitHub

These instructions are for the GitHub.com, not GitHub Enterprise. Use the client ID and secret you created with your GitHub OAuth application in the prerequisites phase ( AWS, GCP).

ii. GitHub Enterprise

The instructions for GitHub Enterprise are similar, with a few extra steps to enable Enterprise and create the required default token.

In the case of GitHub Enterprise add the defaultToken created in the prerequisites phase ( AWS, GCP) to the GitHub section. The hostname should not include the protocol, ex: github.exampleorg.com.

l. Object storage

Regardless of your storage provider, the bucket name you created during the prerequisites phase will need to be included.

object_storage:
  bucketName: "<bucket-name>"

S3 compatible

Add an s3 section as a child of object_storage. The endpoint in the case of AWS S3 is the regional endpoint, it is of the form https://s3.<region>.amazonaws.com. Otherwise it is the API endpoint fo your object storage server.

object_storage:
  ...
  s3:
    enabled: true
    endpoint: "<storage-server-or-s3-endpoint>"

Under object_storage.s3, you may provide the accessKey and secretKey, the irsaRole, or nothing. They were created during the prerequisites steps.

CircleCI server will use the credentials provided to authenticate to S3.

m. Installing behind a proxy

Depending on your security requirements, you might want to install CircleCI server behind a proxy. Installing behind a proxy gives you the power to monitor and control access between your installation and the broader Internet. For further information including limitations of installation behind a proxy, see the Installing server behind a proxy guide.

The following fields need to be configured in your values.yaml:

Toggle proxy.enabled to "1"
Enter details for proxy.http.host and proxy.https.host, along with their associated ports. These values can be the same but they both need to be configured.
For authentication you will need to configure proxy.http.auth.enabled and proxy.https.auth.enabled as "1". You will also need to configure the respective username and password for both HTTP and HTTPS.
configure the no_proxy hosts and subnets. This should include localhost, your GitHub Enterprise host (optional), the hostname of your CircleCI installation (see Known Limitations for an explanation), and the CIDR of Nomad.

proxy:
  enabled: "1"
  http:
    host: "<proxy.example.internal>"
    port: "3128"
    auth:
      enabled: "1"
      username: "<proxy-user>"
      password: "<proxy-password>"
  https:
    host: "<proxy.example.internal>"
    port: "3128"
    auth:
      enabled: "1"
      username: "<proxy-user>"
      password: "<proxy-password>"
  no_proxy:
    - localhost
    - 127.0.0.1
    - "<github.example.internal>"
    - "<circleci.example.internal>"
    - "<nomad-subnet-cidr>"
    - "<vpc-or-subnet-cidr>"   # VPC or subnets to exclude from the proxy (optional)

n. Encrypting environment variables

All environment variables stored in contexts are encrypted using either Google Tink or HashiCorp Vault. We recommend the use of Tink as Vault has been deprecated.

Use Tink

The following steps cover using Tink as an alternative to Vault:

Enable Tink in your values.yaml:
```
tink:
  enabled: true
  keyset: ""
```
When tink.enabled is true, Vault will not be deployed.

Tink or Vault must be set once during installation, and cannot be changed after deployment.
Generate a keyset, which Tink uses to manage key rotation. The easiest way to do this is to use Google’s Tinkey CLI utility. Once installed, use the following command:
```
tinkey create-keyset --key-template XCHACHA20_POLY1305
```
CircleCI server will store your generated keyset in a Kubernetes Secret. You may generate this Secret in either of the following ways:
Option 1: Create the Kubernetes Secret yourself

Following the example below, create a Kubernetes Secret with the name tink and a key keyset. Apply this Secret to the namespace of your CircleCI server installation.

apiVersion: v1 kind: Secret metadata: name: tink data: keyset: <your-keyset>
Option 2: CircleCI server will create the Kubernetes Secret

You may add the keyset to your values.yaml under tink as in the example below. CircleCI will generate the required Secret to store your keyset.

tink: enabled: true keyset: "<your-keyset>"

If your Tink keyset is somehow lost, you will need to generate a new keyset and then recreate your contexts and their associated Secrets.

o. Setting up OIDC (optional)

CircleCI server optionally supports OpenID Connect (OIDC) tokens in jobs.

This feature is not enabled by default and requires a few additional steps to set up, as follows:

Generate a JSON Web Key (JWK) using the default parameters and copy the Public and Private Keypair Set to a secure and convenient location (for example, ~/oidc-service-jwk.json). This key pair will be used by oidc-service to sign the OIDC tokens used in jobs.
The JWK will need to be base64 encoded without line wrapping before using it on server:
- On Linux:
  $ base64 -w0 ~/oidc-service-jwk.json
- On macOS:
  $ base64 -b0 ~/oidc-service-jwk.json
  Copy the output to a secure location, so it can be referenced in the next step.

Finally, enable oidc-service and supply it the base64 encoded JWK from your server Helm values file:

oidc_service:
  isEnabled: true
  json_web_keys: << set your base64 encoded JWK from step 2 here >>

4. Deploy

Once you have completed the fields detailed above, you can deploy CircleCI’s core services:

USERNAME=<provided-username>
PASSWORD=<token>
namespace=<your-namespace>
helm registry login cciserver.azurecr.io/circleci-server -u $USERNAME -p $PASSWORD
helm install circleci-server oci://cciserver.azurecr.io/circleci-server -n $namespace --version 4.7.0 -f <path-to-values.yaml>

5. Create DNS entry

Create a DNS entry for your nginx load balancer, for example, circleci.your.domain.com and app.circleci.your.domain.com. The DNS entry should align with the DNS names used when creating your TLS certificate and GitHub OAuth app during the prerequisites steps. All traffic will be routed through this DNS record.

You need the IP address, or, if using AWS, the DNS name of the nginx load balancer. You can find this information with the following command:

kubectl get service -l app=circleci-proxy

6. Validation

You should now be able to navigate to your CircleCI server installation and log in to the application successfully.

Now we will move on to build services. It may take a while for all your services to be up. You can periodically check by running the following command (you are looking for the frontend pod to show a status of running and ready should show 1/1):

kubectl get pods -n <YOUR_CIRCLECI_NAMESPACE>

Machine provisioner and Nomad server pods are expected to fail at this stage. You will set up your execution environments in the next phase of the installation.

Next steps

Phase 3 AWS: Execution Environments Installation

Suggest an edit to this page

Make a contribution

Learn how to contribute

Still need help?

Ask the CircleCI community

Join the research community

Visit our Support site