CircleCI Server v3.x Operations Overview
The following guide contains information useful for CircleCI Server Operators.
It is assumed that you have already read the Server 3.x Overview.
CircleCI Server schedules CI jobs using the Nomad scheduler. The Nomad control plane runs inside of Kubernetes, while the Nomad clients are provisioned outside the cluster. The Nomad clients need access to the Nomad control plane, output processor, and VM service.
CircleCI Server can run Docker jobs on the Nomad clients, but it can also run jobs in a dedicated VM. These VM jobs are controlled by the Nomad clients, therefore the Nomad clients must be able to access the VM machines on port 22 for SSH and port 2376 for remote Docker jobs.
Job artifacts and output are sent directly from jobs in Nomad to object storage (S3, GCS, or other supported options). Audit logs and other items from the application are also stored in object storage so both the Kubernetes cluster and the Nomad clients will need access to object storage.
Build Environment
CircleCI Server 3.x uses Nomad as the primary job scheduler. Refer to our Introduction to Nomad Cluster Operation to learn more about the job scheduler and how to perform basic client and cluster operations.
By default, CircleCI Nomad clients automatically provision compute resources according to the executors configured for
each job in a project’s .circleci/config.yml file.
Nomad Clients
Nomad Clients run without storing state, enabling you to increase or decrease the number of containers as needed.
To ensure enough Nomad clients are running to handle all builds, track the queued builds and increase the number of Nomad Client machines as needed to balance the load. For more on tracking metrics see the Metrics and Monitoring section.
Each machine reserves two vCPUs and 4GB of memory for coordinating builds. The remaining processors and memory create the containers for running jobs. Larger machines are able to run more containers and are limited by the number of available cores after two are reserved for coordination.
| The maximum machine size for a Nomad client is 128GB RAM/64 CPUs. Contact your CircleCI account representative to request use of larger machines for Nomad Clients. |
For more information on Nomad port requirements, see the Hardening Your Cluster section.
GitHub
CircleCI uses GitHub or GitHub Enterprise as an identity provider. GitHub Enterprise can, in turn, use SAML or SCIM to manage users from an external identity provider.
| CircleCI does not support changing the URL or back-end GitHub instance after it has been set up. |
The following table describes the ports used on machines running GitHub to communicate with the services and Nomad Client instances.
| Source | Ports | Use |
|---|---|---|
Services |
22 |
Git access |
Services |
80 or 443 |
API access |
Nomad Client |
22 |
Git access |
Nomad Client |
80 or 443 |
API access |