System Requirements
This document is intended for system administrators of self-hosted installations of CircleCI Server.
This section defines the system and port access requirements for installing CircleCI v2.19.5.
Services Machine
The Services machine hosts the core of our Server product, including the user-facing website, API engine, datastores, and Nomad job scheduler. It is best practice to use an isolated machine.
The following table defines the Services machine CPU, RAM, and disk space requirements:
| Number of daily active CircleCI users | CPU | RAM | Disk space | NIC speed |
|---|---|---|---|---|
<50 |
8 cores |
32GB |
100GB |
1Gbps |
50-250 |
12 cores |
64GB |
200GB |
1Gbps |
251-1000 |
16 cores |
128GB |
500GB |
10Gbps |
1001-5000 |
20 cores |
256GB |
1TB |
10Gbps |
5000+ |
24 cores |
512GB |
2TB |
10Gbps |
Nomad Clients
Nomad client machines run CircleCI jobs that are scheduled by the Nomad Server. The Minimum CPU, RAM, and disk space requirements per client are as follows:
-
CPU: 4 cores
-
RAM: 32GB
-
Disk space: 100GB
-
NIC speed: 1Gbps
For an AWS install of CircleCI Server, the recommended instance type for Nomad clients is m5.2xlarge (8 vCPUs @ 2.4GHz, 32GB RAM).
You can choose a larger instance type to fit more jobs per Client. To help in this choice, consider that when Nomad decides if a job will fit on a Client, the Job is allocated 1024MHz per CPU, and capacity is number of cores * clock speed. Using this method, the maximum capacity of a m5.2xlarge would be 19200MHz, which would mean 9.6 jobs could run on that client (if there were no limiting factors). In practice, Nomad will researve some CPU for itself, and because of the CPU:RAM ratio, the available RAM is the limiting factor governing how many jobs can run.
Recommendations for Heavy Usage
EBS will throttle IOPS after certain IO operations, therefore heavy-load usage with the default Server configuration (m5.2xlarge with 200-GiB gp2 root volume) can see performance-related issues, such as slow jobs, slow spin-up, and job timeouts in worst cases.
To give an idea of what is meant by heavy usage:
-
If
BurstBalancemetrics for storage attached to Nomad clients are decreasing sharply, this is a sign that the workload is heavy enough to be affected. See the AWS EBS metrics document for further details. -
We have had a report from a customer that continuous disk I/O at 500 MB/s used up the burst credits for our default storage configuration within 37.5 minutes.
Suggested workarounds for this issue are as follows:
-
Use EBS volumes larger than 1 TiB. Large EBS volumes never consume IO credits.
-
Use the *5d instance type and use an attached instance store for Docker-related files (i.e. /var/lib/docker).
Choosing Nomad Client Quantity
The following table defines the number of Nomad clients to make available as a best practice. Scale up and down according to demand on your system:
| Number of daily active CircleCI users | Number of Nomad client machines |
|---|---|
<50 |
1-5 |
50-250 |
5-10 |
250-1000 |
10-15 |
5000+ |
15+ |
Server Ports
Below all ports required by a CircleCI 2.0 installation are listed for each machine type.
Services Machine
| Port number | Protocol | Direction | Source / destination | Use | Notes |
|---|---|---|---|---|---|
80 |
TCP |
Inbound |
End users |
HTTP web app traffic |
|
443 |
TCP |
Inbound |
End users |
HTTPS web app traffic |
|
7171 |
TCP |
Inbound |
End users |
Artifacts access |
|
8081 |
TCP |
Inbound |
End users |
Artifacts access |
|
22 |
TCP |
Inbound |
Administrators |
SSH |
|
8800 |
TCP |
Inbound |
Administrators |
Admin console |
|
8125 |
UDP |
Inbound |
Nomad Clients |
Metrics |
|
8125 |
UDP |
Inbound |
Nomad Servers |
Metrics |
Only if using externalized Nomad Servers |
8125 |
UDP |
Inbound |
All Database Servers |
Metrics |
Only if using externalised databases |
4647 |
TCP |
Bi-directional |
Nomad Clients |
Internal communication |
|
8585 |
TCP |
Bi-directional |
Nomad Clients |
Internal communication |
|
7171 |
TCP |
Bi-directional |
Nomad Clients |
Internal communication |
|
3001 |
TCP |
Bi-directional |
Nomad Clients |
Internal communication |
|
80 |
TCP |
Bi-directional |
GitHub Enterprise / GitHub.com (whichever applies) |
Webhooks / API access |
|
443 |
TCP |
Bi-directional |
GitHub Enterprise / GitHub.com (whichever applies) |
Webhooks / API access |
|
80 |
TCP |
Outbound |
AWS API endpoints |
API access |
Only if running on AWS |
443 |
TCP |
Outbound |
AWS API endpoints |
API access |
Only if running on AWS |
5432 |
TCP |
Outbound |
PostgreSQL Servers |
PostgreSQL database connection |
Only if using externalised databases. Port is user-defined, assuming the default PostgreSQL port. |
27017 |
TCP |
Outbound |
MongoDB Servers |
MongoDB database connection |
Only if using externalized databases. Port is user-defined, assuming the default MongoDB port. |
5672 |
TCP |
Outbound |
RabbitMQ Servers |
RabbitMQ connection |
Only if using externalized RabbitMQ |
6379 |
TCP |
Outbound |
Redis Servers |
Redis connection |
Only if using externalized Redis |
4647 |
TCP |
Outbound |
Nomad Servers |
Nomad Server connection |
Only if using externalized Nomad Servers |
443 |
TCP |
Outbound |
CloudWatch Endpoints |
Metrics |
Only if using AWS CloudWatch |
Nomad Clients
| Port number | Protocol | Direction | Source / destination | Use | Notes |
|---|---|---|---|---|---|
64535-65535 |
TCP |
Inbound |
End users |
SSH into builds feature |
|
80 |
TCP |
Inbound |
Administrators |
CircleCI Admin API access |
|
443 |
TCP |
Inbound |
Administrators |
CircleCI Admin API access |
|
22 |
TCP |
Inbound |
Administrators |
SSH |
|
22 |
TCP |
Outbound |
GitHub Enterprise / GitHub.com (whichever applies) |
Download Code From GitHub. |
|
4647 |
TCP |
Bi-directional |
Services Machine |
Internal communication |
|
8585 |
TCP |
Bi-directional |
Services Machine |
Internal communication |
|
7171 |
TCP |
Bi-directional |
Services Machine |
Internal communication |
|
3001 |
TCP |
Bi-directional |
Services Machine |
Internal communication |
|
443 |
TCP |
Outbound |
Cloud Storage Provider |
Artifacts storage |
Only if using external artifacts storage |
53 |
UDP |
Outbound |
Internal DNS Server |
DNS resolution |
This is to make sure that your jobs can resolve all DNS names that are needed for their correct operation. |
GitHub Enterprise / GitHub.com
| Port number | Protocol | Direction | Source / destination | Use | Notes |
|---|---|---|---|---|---|
22 |
TCP |
Inbound |
Services Machine |
Git access |
|
22 |
TCP |
Inbound |
Nomad Clients |
Git access |
|
80 |
TCP |
Inbound |
Nomad Clients |
API access |
|
443 |
TCP |
Inbound |
Nomad Clients |
API access |
|
80 |
TCP |
Bi-directional |
Services Machine |
Webhooks / API access |
PostgreSQL Servers
| Port number | Protocol | Direction | Source / destination | Use | Notes |
|---|---|---|---|---|---|
5432 |
TCP |
Bi-directional |
PostgreSQL Servers |
PostgreSQL replication |
Only if using externalized databases. Port is user-defined, assuming the default PostgreSQL port. |
MongoDB Servers
| Port number | Protocol | Direction | Source / destination | Use | Notes |
|---|---|---|---|---|---|
27017 |
TCP |
Bi-directional |
MongoDB Servers |
MongoDB replication |
Only if using externalized databases. Port is user-defined, assuming the default MongoDB port. |
RabbitMQ Servers
| Port number | Protocol | Direction | Source / destination | Use | Notes |
|---|---|---|---|---|---|
5672 |
TCP |
Inbound |
Services Machine |
RabbitMQ connection |
Only if using externalized RabbitMQ |
5672 |
TCP |
Bi-directional |
RabbitMQ Servers |
RabbitMQ mirroring |
Only if using externalized RabbitMQ |
Redis Servers
| Port number | Protocol | Direction | Source / destination | Use | Notes |
|---|---|---|---|---|---|
6379 |
TCP |
Inbound |
Services Machine |
Redis connection |
Only if using externalized Redis |
6379 |
TCP |
Bi-directional |
Redis Servers |
Redis replication |
Only if using externalized Redis, and using Redis replication (optional) |
Nomad Servers
| Port number | Protocol | Direction | Source / destination | Use | Notes |
|---|---|---|---|---|---|
4646 |
TCP |
Inbound |
Services Machine |
Nomad Server connection |
Only if using externalized Nomad Servers |
4647 |
TCP |
Inbound |
Services Machine |
Nomad Server connection |
Only if using externalized Nomad Servers |
4648 |
TCP |
Bi-directional |
Nomad Servers |
Nomad Servers internal communication |
Only if using externalized Nomad Servers |