> For the complete documentation index, see [llms.txt](https://circleci.com/docs/llms.txt) # CircleCI Server 4.9 overview ## Introduction CircleCI Server is an on-premises CI/CD platform for enterprise customers. Server is suited for organizations who have compliance or security needs that require them to operate within their firewall, in a private cloud, or in a data center. CircleCI Server provides the same features as CircleCI’s cloud offering, but operates within your Kubernetes cluster. ## Architecture The following diagram presents and overview of the services architecture of CircleCI Server. Figure 1. CircleCI Server 4.9 Architecture ### Ports and services The CircleCI Server application exposes two services, `nginx` and `nomad-server`, using a single load balancer. If required, the load balancer can be made private, separating it from the public internet. Service Ports Description Frontend GUI Proxy & API 80 and 443 Exposes the web application. Nomad Control Plane 4647 Exposes an RPC protocol for Nomad clients. The application exposes a number of external ports. These ports are used for various functions as defined in the table below. Port number Protocol Direction Source / Destination Use Notes `80` TCP Inbound End users HTTP web app traffic `443` TCP Inbound End users HTTP web app traffic `22` TCP Inbound Administrators SSH Only required for the bastion host. `64535`\-`65535` TCP Inbound SSH into builds Only required for the Nomad clients. ### Job scheduling and artifact storage CircleCI Server schedules CI jobs using HashiCorp [Nomad](https://www.nomadproject.io/). Depending on the execution environment used for a job, the resources are provisioned through one of the following: * Nomad for the Docker executor. * Machine provisioner for virtual machines (VMs), the `machine` executor. * Runner, if used, for all executor types. The Nomad control plane runs inside your Kubernetes cluster. Nomad clients, which are responsible for running scheduled CircleCI jobs, are provisioned outside the cluster. CircleCI Server can run Docker jobs on the Nomad clients themselves or in a dedicated virtual machine (VM). For more information on Nomad architecture and use, see the [Introduction to Nomad Cluster Operation](https://circleci.com/docs/server-admin/server-4.9/operator/introduction-to-nomad-cluster-operation/). Job artifacts and outputs are sent directly from jobs in Nomad to object storage (S3, Google Cloud Storage, or other supported options). Audit logs and other items from the application are also stored in object storage, so both the Kubernetes cluster and the Nomad clients need access to object storage. ## List of services and their function CircleCI Server 4.9 consists of the following services. Find their descriptions and failure implications below: Service Component Description What happens if it fails? Notes `api-service` App Core Provides a GraphQL API that provides data to render the web frontend. Many parts of the UI (such as Contexts) will fail completely. `audit-log-service` App Core Persists audit log events to blob storage for long-term storage. Some events may not be recorded. `branch-service` App Core A service responsible for listening to the event stream. Detects branch deletions, job updates, pushes, workflow updates. `builds-service` App Core Ingests from `www-api` and sends to plans-service, workflows-conductor, and to orbs-service. `circleci-mongodb` Execution Primary datastore `circleci-postgres` Data storage for microservices. `circleci-rabbitmq` Pipelines and Execution Queuing for workflow messaging, test results, usage, cron tasks, output, notifications, and scheduler. `circleci-redis` Execution Caches data that will not be stored permanently (such as build logs), for request caching, and for rate limit calculations. A failed cache can result in rate limiting from the VCS if too many calls are made to it. `circleci-telegraf` Telegraf collects statsd metrics. All white-boxed metrics in our services publish statsd metrics that are sent to Telegraf, but can also be configured to be exported to other places (such as Datadog or Prometheus). `circleci-vault` HashiCorp Vault to run encryption and decryption as a service for secrets. `contexts-service` App Core Stores and provides encrypted contexts. All builds using Contexts will fail. `cron-service` Pipelines Triggers scheduled workflows. Scheduled workflows will not run. `distributor-*` App Core Responsible for accepting build requests and distributing the job to appropriate queues. `docker-provisioner-*` Docker Compute Management Responsible for scheduling jobs on Nomad cluster(s). `domain-service` App Core Stores and provides information about our domain model. Works with permissions and API. Workflows will fail to start and some REST API calls may fail, causing 500 errors in the CircleCI UI. If LDAP authentication is in use, all logins will fail. `execution-gateway-*` Execution Boundary for execution to provide API to rest of CircleCI `feature-flags-api` Execution Used to configure and fetch feature flags. Systems will use defaults in place of any flags. In the case of server it has no effect, since no feature flags will be set. `frontend` Frontend CircleCI web app and `www-api` proxy. The UI and REST API will be unavailable and no jobs will be triggered by GitHub/Enterprise. Running builds will be OK, but no updates will be seen. Rate limit of 150 requests per second with a single user instantaneous limit of 300 requests. `insights-service` Metrics A service to aggregate build and usage metrics for exporting and analysis. `kong` App Core API management. `legacy-notifier` App Core Handles notifications to external services (for example, Slack or email). `machine-provisioner-*` Machine Compute Management Periodically requests machines from compute provider to run jobs for both `machine` and remote Docker. Periodically checks for stale machine and remote Docker instances and requests that provider removes them. `nginx` App Core / Frontend Handles traffic redirection and ingress. `nomad-autoscaler` Nomad Manages scaling of Nomad clusters in AWS and GCP environments. `nomad-server` Nomad Responsible for managing Nomad Clients. `oidc-service` App Core Mints OIDC tokens, serves OpenID configuration and public JSON web keys. OIDC tokens would not be present in jobs. Jobs depending on images present in AWS ECR and authenticated with OIDC will fail to run. `oidc-tasks-service` App Core Provides interface for minting OIDC tokens with customizable claims. Integrates with the OIDC service. OIDC tokens would not be present in jobs. Jobs depending on images present in AWS ECR and authenticated with OIDC will fail to run. `orb-service` Pipelines Handles communication between orb registry and config. output Execution Receives job output and status updates and writes them to S3. Also provides an API to running jobs to access caches, workspaces, store caches, workspaces, artifacts, and test results. `permissions-service` App Core Provides the CircleCI permissions interface. Workflows will fail to start and some REST API calls may fail, causing 500 errors in the UI. `policy-service-*` App Core Core service of config policies framework. Allows management of policy documents and policy bundles, and evaluates inputs against these bundles using the `circle-policy-agent`. While config policies is enabled for your organization, all pipelines will fail to run. `runner-admin` Runner Admin Manages Runner resource classes and tokens, & coordinates runner task handling `soketi` Frontend WebSockets server. `step` Frontend Provides UI output in the job view `telegraf` Metrics Collection of metrics. `web-ui-*` Frontend Micro Front End (MFE) services used to render the frontend web application GUI. The respective services page will fail to load. Example: A `web-ui-server-admin` failure means the server Admin page will fail to load. The MFEs are used to render the web application located at `app.` `webhook-service` App Core Service responsible for all webhooks, including management of state and handling events. `workflows-conductor-event-consumer` Pipelines Takes in information from VCS to kick off pipelines. New Pipelines will not be kicked off when there are changes in the VCS. `workflows-conductor-grpc` Pipelines Helps translate the information through gRPC. ## Supported platforms CircleCI Server is designed to deploy within a Kubernetes cluster. The machine service (machine provisioner) is able to leverage unique EKS or GKE offerings to dynamically create VM images. If installing outside of EKS or GKE, additional work is required to access some of the same machine build features. Setting up CircleCI runner gives you access to the same feature set as machine provisioner across a much wider range of operating systems and machine types (for example, macOS). We do our best to support a wide range of platforms for installation. We use environment-agnostic solutions wherever possible. However, we do not test all platforms and options. For that reason, we provide a list of tested environments, which we will continue to expand. Environment Status Notes EKS Tested GKE Tested Local installation Tested Azure Untested Should work with S3 compatible object store and Runner. Digital Ocean Untested Should work with S3 compatible object store and Runner. OpenShift Untested Known not to work. Rancher Untested Should work with S3 compatible object store and Runner. ## Installation options CircleCI Server can be installed using cloud resources (GCP or AWS), locally, and in an air-gapped environment. For installation instructions, refer to the following: * Cloud installation guides for [AWS](https://circleci.com/docs/server-admin/server-4.9/installation/phase-1-aws-prerequisites/) and [GCP](https://circleci.com/docs/server-admin/server-4.9/installation/phase-1-gcp-prerequisites/). Follow the sections for your chosen cloud provider. * [Local](https://circleci.com/docs/server-admin/server-4.9/installation/phase-1-aws-prerequisites/) installation. Notes are provided in the main installation guide to show which sections are optional, or different, for local installations. * [Air-gapped](https://circleci.com/docs/server-admin/server-4.9/air-gapped-installation/phase-1-prerequisites/) installation guide. This guide is designed to be used along-side key sections of the main installation guides for [AWS](https://circleci.com/docs/server-admin/server-4.9/installation/phase-1-aws-prerequisites/) and [GCP](https://circleci.com/docs/server-admin/server-4.9/installation/phase-1-gcp-prerequisites/). ## Next steps * [CircleCI Server 4.9 Release Notes](https://circleci.com/docs/server-admin/server-4.9/overview/release-notes/). * server 4.9 installation prerequisites for [AWS](https://circleci.com/docs/server-admin/server-4.9/installation/phase-1-aws-prerequisites/) and [GCP](https://circleci.com/docs/server-admin/server-4.9/installation/phase-1-gcp-prerequisites/).