Phase 2 - core services
On This Page
- 1. Create a namespace
- 2. Pull images
- 3. Create Helm values
- a. API token
- b. Session cookie
- c. Encryption
- d. PostgreSQL
- Credentials
- TLS
- e. MongoDB credentials
- f. RabbitMQ configurations and auth Secrets
- g. Pusher Kubernetes Secret
- h. Global
- CircleCI domain name (required)
- License
- Registry
- i. Static IP addresses
- GCP - add static IP address
- AWS - add Elastic IP addresses
- j. TLS
- k. GitHub integration
- GitHub
- GitHub Enterprise
- l. Object storage
- S3 compatible
- Google Cloud Storage
- m. Installing behind a proxy
- n. Encrypting environment variables
- Use Tink
- o. Setting up OIDC (optional)
- 4. Deploy
- 5. Create DNS entry
- 6. Validation
- Next steps
Before you begin with the CircleCI server v4.5 core services installation phase, ensure all prerequisites are met.
In the following sections, replace any sections indicated by < > with your details. |
1. Create a namespace
Create a namespace to install the application into.
kubectl create ns <namespace>
Once you have created your namespace, we recommend setting your kubectl context too, with the following command: kubectl config set-context --current --namespace <namespace> |
2. Pull images
Credentials to pull the images from CircleCI’s image registry will be provided to you as part of the onboarding process. A docker-registry
Kubernetes Secret will be used to pull images from Azure Container Registry (ACR). You have two options, depending on whether your application has access to the public internet.
3. Create Helm values
Before installing CircleCI, it is recommended to create a new values.yaml
file unique to your installation. The Installation Reference section contains some example values.yaml
files that are a good place to start. The following describes the minimum required values to include in values.yaml
. Additional customizations are available, see the provided values.yaml
for all available options.
For sensitive data there are two options:
-
Add into the
values.yaml
file -
Add them as Kubernetes Secrets directly
This flexibility allows you to manage Kubernetes Secrets using whichever process you prefer. Whichever option you choose, this sensitive information is stored as a Kubernetes Secret within CircleCI.
During the installation process, you may use the following command to generate a random alphanumeric value as required: cat /dev/urandom | LC_ALL=C tr -dc 'a-zA-Z0-9' | head -c <num-of-chars> . This command should work on any *nix based system. |
a. API token
The application requires a Kubernetes Secret containing an API token. This API token is used to facilitate internal API communication to api-service. Use a random string and store it securely. CircleCI will not be able to recover this value if lost. There are two options depending on whether you want to create the Kubernetes Secret, or if you want CircleCI to create it for you.
b. Session cookie
The application requires a session cookie key Kubernetes Secret, which CircleCI uses to sign session cookies. The Secret must be exactly 16 characters long. Use a random string and store it securely. CircleCI will not be able to recover this value if lost. There are two options depending on whether you want to create the Kubernetes Secret, or if you want CircleCI to create it for you.
c. Encryption
The application requires a Kubernetes Secret containing signing and encryption keysets. These keysets are used to encrypt and sign artifacts generated by CircleCI. These keys were created during the prerequisites phase. CircleCI will not be able to recover the values if lost. Depending on how you prefer to manage Kubernetes Secrets, there are two options.
d. PostgreSQL
Credentials
The application requires a Kubernetes Secret containing PostgreSQL credentials. This is true when using either the internal (default) or an externally hosted instance of PostgreSQL. CircleCI will not be able to recover the values if lost. Based on how you prefer to manage Kubernetes Secrets there are two options.
TLS
PostgreSQL may be extended to use TLS encrypted traffic. When deployed internally, this option is disabled by default but may be enabled by adding the following to your PostgreSQL block of your values.yaml
postgresql:
...
tls:
enabled: true
autoGenerated: true # Generate automatically self-signed TLS certificates
Certificate files may also be provided, rather than autogenerated. In this case, create a Secret containing the TLS certs and keys needed.
kubectl -n <namespace> create secret generic postgres-tls-secret --from-file=./cert.pem --from-file=./cert.key --from-file=./ca.pem
Then the PostgreSQL block in your values.yaml
will contain the contents below.
postgresql:
...
tls:
enabled: true
certificatesSecret: "postgres-tls-secret" # Name of an existing secret that contains the certificates
certFilename: "cert.pem" # Certificate filename
certKeyFilename: "cert.key" # Certificate key filename
certCAFilename: "ca.pem" # CA Certificate filename
e. MongoDB credentials
The application requires a Kubernetes Secret containing MongoDB credentials. This is true when using either the internal (default) or an externally hosted instance of MongoDB. CircleCI will not be able to recover the values if lost. Based on how you prefer to manage Kubernetes Secrets there are two options.
f. RabbitMQ configurations and auth Secrets
The RabbitMQ installation requires two random alphanumeric strings. CircleCI will not be able to recover the values if lost. Based on how you prefer to manage Kubernetes Secrets there are two options.
g. Pusher Kubernetes Secret
The application requires a Kubernetes Secret for Pusher. CircleCI will not be able to recover the values if lost. Based on how you prefer to manage Kubernetes Secrets there are 2 options:
h. Global
All values in this section are children of global
in your values.yaml
.
CircleCI domain name (required)
Enter the domain name you specified when creating your Frontend TLS key and certificate.
global:
...
domainName: "<full-domain-name-of-your-install>"
License
A license will be provided by CircleCI, add it to values.yaml
:
global:
...
license: '<license>'
Registry
The registry to pull images from will have been provided to you, or you may have added the images to your own hosted registry. Add the registry to values.yaml
:
global:
...
container:
registry: <registry-domain eg: cciserver.azurecr.io >
org: <your-org-if-applicable>
i. Static IP addresses
If you provisioned a Static IP (GCP) or Elastic IP addresses (AWS) in the prerequisites, you can now add the values under the nginx block.
GCP - add static IP address
For GCP, add the provisioned IPv4 address under the loadBalancerIp
field in the nginx block.
nginx:
...
loadBalancerIp: "<gcp-provisioned-ipv4-address>"
AWS - add Elastic IP addresses
For AWS, under nginx annotations, add the service.beta.kubernetes.io/aws-load-balancer-eip-allocations
annotation with each of the AllocationId
values generated as a comma separated list. The number of `AllocationId`s must match the number of subnets the load balancer is deployed into (default 3).
nginx:
...
annotations:
...
service.beta.kubernetes.io/aws-load-balancer-eip-allocations: <eip-id-1>,<eip-id-2>,<eip-id-3>
j. TLS
For TLS, you have 4 options:
k. GitHub integration
To configure GitHub with CircleCI, there are two options for providing credentials to the deployment. Steps for both GitHub and GitHub Enterprise (GHE) are given in the next two sections.
GitHub
These instructions are for the GitHub.com, not GitHub Enterprise. Use the client ID and secret you created with your GitHub OAuth application in the prerequisites phase.
GitHub Enterprise
The instructions for GitHub Enterprise are similar, with a few extra steps to enable Enterprise and create the required default token.
In the case of GitHub Enterprise add the defaultToken
created in the prerequisite phase to the GitHub
section. The hostname should not include the protocol, ex: github.exampleorg.com
.
l. Object storage
Regardless of your storage provider, the bucket name you created during the prerequisites phase will need to be included.
object_storage:
bucketName: "<bucket-name>"
S3 compatible
Add an s3
section as a child of object_storage
. The endpoint
in the case of AWS S3 is the regional endpoint, it is of the form https://s3.<region>.amazonaws.com
. Otherwise it is the API endpoint fo your object storage server.
object_storage:
...
s3:
enabled: true
endpoint: "<storage-server-or-s3-endpoint>"
Under object_storage.s3
, you may provide the accessKey
and secretKey
, the irsaRole
, or nothing. They were created during the prerequisites steps.
CircleCI server will use the credentials provided to authenticate to S3.
Google Cloud Storage
Under object_storage
add the following.
gcs:
enabled: true
Under object_storage.gcs
you may add service_account
, workloadIdentity
, or neither. The keys/role were created during the prerequisites steps.
m. Installing behind a proxy
Depending on your security requirements, you might want to install CircleCI server behind a proxy. Installing behind a proxy gives you the power to monitor and control access between your installation and the broader Internet. For further information including limitations of installation behind a proxy, see the Installing server behind a proxy guide.
The following fields need to be configured in your values.yaml
:
-
Toggle
proxy.enabled
to"1"
-
Enter details for
proxy.http.host
andproxy.https.host
, along with their associated ports. These values can be the same but they both need to be configured. -
For authentication you will need to configure
proxy.http.auth.enabled
andproxy.https.auth.enabled
as"1"
. You will also need to configure the respective username and password for both HTTP and HTTPS. -
configure the
no_proxy
hosts and subnets. This should include localhost, your GitHub Enterprise host (optional), the hostname of your CircleCI installation (see Known Limitations for an explanation), and the CIDR of Nomad.
proxy:
enabled: "1"
http:
host: "<proxy.example.internal>"
port: "3128"
auth:
enabled: "1"
username: "<proxy-user>"
password: "<proxy-password>"
https:
host: "<proxy.example.internal>"
port: "3128"
auth:
enabled: "1"
username: "<proxy-user>"
password: "<proxy-password>"
no_proxy:
- localhost
- 127.0.0.1
- "<github.example.internal>"
- "<circleci.example.internal>"
- "<nomad-subnet-cidr>"
- "<vpc-or-subnet-cidr>" # VPC or subnets to exclude from the proxy (optional)
n. Encrypting environment variables
All environment variables stored in contexts are encrypted using either Google Tink or HashiCorp Vault. We recommend the use of Tink as Vault has been deprecated.
Use Tink
The following steps cover using Tink as an alternative to Vault:
-
Enable Tink in your
values.yaml
:tink: enabled: true keyset: ""
When
tink.enabled
is true, Vault will not be deployed.Tink or Vault must be set once during installation, and cannot be changed after deployment. -
Generate a keyset, which Tink uses to manage key rotation. The easiest way to do this is to use Google’s Tinkey CLI utility. Once installed, use the following command:
tinkey create-keyset --key-template XCHACHA20_POLY1305
-
CircleCI server will store your generated keyset in a Kubernetes Secret. You may generate this Secret in either of the following ways:
If your Tink keyset is somehow lost, you will need to generate a new keyset and then recreate your contexts and their associated Secrets. |
o. Setting up OIDC (optional)
CircleCI server optionally supports OpenID Connect (OIDC) tokens in jobs.
This feature is not enabled by default and requires a few additional steps to set up, as follows:
-
Generate a JSON Web Key (JWK) using the default parameters and copy the
Public and Private Keypair Set
to a secure and convenient location (for example,~/oidc-service-jwk.json
). This key pair will be used byoidc-service
to sign the OIDC tokens used in jobs. -
The JWK will need to be base64 encoded without line wrapping before using it on server:
-
On Linux:
$ base64 -w0 ~/oidc-service-jwk.json
-
On macOS:
$ base64 -b0 -i ~/oidc-service-jwk.json
Copy the output to a secure location, so it can be referenced in the next step.
-
-
Finally, enable
oidc-service
and supply it the base64 encoded JWK from your server Helm values file:oidc_service: isEnabled: true json_web_keys: << set your base64 encoded JWK from step 2 here >>
4. Deploy
Once you have completed the fields detailed above, you can deploy CircleCI’s core services:
USERNAME=<provided-username>
PASSWORD=<token>
namespace=<your-namespace>
helm registry login cciserver.azurecr.io/circleci-server -u $USERNAME -p $PASSWORD
helm install circleci-server oci://cciserver.azurecr.io/circleci-server -n $namespace --version 4.5.3 -f <path-to-values.yaml>
5. Create DNS entry
Create a DNS entry for your nginx load balancer, for example, circleci.your.domain.com
and app.circleci.your.domain.com
. The DNS entry should align with the DNS names used when creating your TLS certificate and GitHub OAuth app during the prerequisites steps. All traffic will be routed through this DNS record.
You need the IP address, or, if using AWS, the DNS name of the nginx load balancer. You can find this information with the following command:
kubectl get service circleci-proxy
6. Validation
You should now be able to navigate to your CircleCI server installation and log in to the application successfully.
Now we will move on to build services. It may take a while for all your services to be up. You can periodically check by running the following command (you are looking for the frontend
pod to show a status of running
and ready should show 1/1
):
kubectl get pods -n <YOUR_CIRCLECI_NAMESPACE>
Machine provisioner and Nomad server pods are expected to fail at this stage. You will set up your execution environments in the next phase of the installation. |