System Requirements
This section defines the system and port access requirements for installing CircleCI v2.19.11.
Services Machine
The Services machine hosts the core of our Server product, including the user-facing website, API engine, datastores, and Nomad job scheduler. It is best practice to use an isolated machine.
Currently, CircleCI Server only supports x86_64 architecture. |
The following table defines the Services machine CPU, RAM, and disk space requirements:
Number of daily active CircleCI users | CPU | RAM | Disk space | NIC speed |
---|---|---|---|---|
<50 |
8 cores |
32GB |
100GB |
1Gbps |
50-250 |
12 cores |
64GB |
200GB |
1Gbps |
251-1000 |
16 cores |
128GB |
500GB |
10Gbps |
1001-5000 |
20 cores |
256GB |
1TB |
10Gbps |
5000+ |
24 cores |
512GB |
2TB |
10Gbps |
Nomad Clients
Nomad client machines run CircleCI jobs that are scheduled by the Nomad Server. The Minimum CPU, RAM, and disk space requirements per client are as follows:
-
CPU: 4 cores
-
RAM: 32GB
-
Disk space: 100GB
-
NIC speed: 1Gbps
For an AWS install of CircleCI Server, the recommended instance type for Nomad clients is m5.2xlarge
(8 vCPUs @ 2.4GHz, 32GB RAM).
Currently, Nomad Clients only support x86_64 architecture. |
You can choose a larger instance type to fit more jobs per Client. To help in this choice, consider that when Nomad decides if a job will fit on a Client, the Job is allocated 1024MHz per CPU, and capacity is number of cores
* clock speed
. Using this method, the maximum capacity of a m5.2xlarge
would be 19200MHz
, which would mean 9.6 jobs could run on that client (if there were no limiting factors). In practice, Nomad will researve some CPU for itself, and because of the CPU:RAM ratio, the available RAM is the limiting factor governing how many jobs can run.
Recommendations for Heavy Usage
EBS will throttle IOPS after certain IO operations, therefore heavy-load usage with the default Server configuration (m5.2xlarge with 200-GiB gp2 root volume) can see performance-related issues, such as slow jobs, slow spin-up, and job timeouts in worst cases.
To give an idea of what is meant by heavy usage:
-
If
BurstBalance
metrics for storage attached to Nomad clients are decreasing sharply, this is a sign that the workload is heavy enough to be affected. See the AWS EBS metrics document for further details. -
We have had a report from a customer that continuous disk I/O at 500 MB/s used up the burst credits for our default storage configuration within 37.5 minutes.
Suggested workarounds for this issue are as follows:
-
Use EBS volumes larger than 1 TiB. Large EBS volumes never consume IO credits.
-
Use the *5d instance type and use an attached instance store for Docker-related files (i.e. /var/lib/docker).
Choosing Nomad Client Quantity
The following table defines the number of Nomad clients to make available as a best practice. Scale up and down according to demand on your system:
Number of daily active CircleCI users | Number of Nomad client machines |
---|---|
<50 |
1-5 |
50-250 |
5-10 |
250-1000 |
10-15 |
5000+ |
15+ |
Server Ports
Below all ports required by a CircleCI 2.0 installation are listed for each machine type.
Services Machine
Port number | Protocol | Direction | Source / destination | Use | Notes |
---|---|---|---|---|---|
80 |
TCP |
Inbound |
End users |
HTTP web app traffic |
|
443 |
TCP |
Inbound |
End users |
HTTPS web app traffic |
|
7171 |
TCP |
Inbound |
End users |
Artifacts access |
|
8081 |
TCP |
Inbound |
End users |
Artifacts access |
|
22 |
TCP |
Inbound |
Administrators |
SSH |
|
8800 |
TCP |
Inbound |
Administrators |
Admin console |
|
8125 |
UDP |
Inbound |
Nomad Clients |
Metrics |
|
8125 |
UDP |
Inbound |
Nomad Servers |
Metrics |
Only if using externalized Nomad Servers |
8125 |
UDP |
Inbound |
All Database Servers |
Metrics |
Only if using externalised databases |
4647 |
TCP |
Bi-directional |
Nomad Clients |
Internal communication |
|
8585 |
TCP |
Bi-directional |
Nomad Clients |
Internal communication |
|
7171 |
TCP |
Bi-directional |
Nomad Clients |
Internal communication |
|
3001 |
TCP |
Bi-directional |
Nomad Clients |
Internal communication |
|
80 |
TCP |
Bi-directional |
GitHub Enterprise / GitHub.com (whichever applies) |
Webhooks / API access |
|
443 |
TCP |
Bi-directional |
GitHub Enterprise / GitHub.com (whichever applies) |
Webhooks / API access |
|
80 |
TCP |
Outbound |
AWS API endpoints |
API access |
Only if running on AWS |
443 |
TCP |
Outbound |
AWS API endpoints |
API access |
Only if running on AWS |
5432 |
TCP |
Outbound |
PostgreSQL Servers |
PostgreSQL database connection |
Only if using externalised databases. Port is user-defined, assuming the default PostgreSQL port. |
27017 |
TCP |
Outbound |
MongoDB Servers |
MongoDB database connection |
Only if using externalized databases. Port is user-defined, assuming the default MongoDB port. |
5672 |
TCP |
Outbound |
RabbitMQ Servers |
RabbitMQ connection |
Only if using externalized RabbitMQ |
6379 |
TCP |
Outbound |
Redis Servers |
Redis connection |
Only if using externalized Redis |
4647 |
TCP |
Outbound |
Nomad Servers |
Nomad Server connection |
Only if using externalized Nomad Servers |
443 |
TCP |
Outbound |
CloudWatch Endpoints |
Metrics |
Only if using AWS CloudWatch |
Nomad Clients
Port number | Protocol | Direction | Source / destination | Use | Notes |
---|---|---|---|---|---|
64535-65535 |
TCP |
Inbound |
End users |
SSH into builds feature |
|
80 |
TCP |
Inbound |
Administrators |
CircleCI Admin API access |
|
443 |
TCP |
Inbound |
Administrators |
CircleCI Admin API access |
|
22 |
TCP |
Inbound |
Administrators |
SSH |
|
22 |
TCP |
Outbound |
GitHub Enterprise / GitHub.com (whichever applies) |
Download Code From GitHub. |
|
4647 |
TCP |
Bi-directional |
Services Machine |
Internal communication |
|
8585 |
TCP |
Bi-directional |
Services Machine |
Internal communication |
|
7171 |
TCP |
Bi-directional |
Services Machine |
Internal communication |
|
3001 |
TCP |
Bi-directional |
Services Machine |
Internal communication |
|
443 |
TCP |
Outbound |
Cloud Storage Provider |
Artifacts storage |
Only if using external artifacts storage |
53 |
UDP |
Outbound |
Internal DNS Server |
DNS resolution |
This is to make sure that your jobs can resolve all DNS names that are needed for their correct operation. |
GitHub Enterprise / GitHub.com
Port number | Protocol | Direction | Source / destination | Use | Notes |
---|---|---|---|---|---|
22 |
TCP |
Inbound |
Services Machine |
Git access |
|
22 |
TCP |
Inbound |
Nomad Clients |
Git access |
|
80 |
TCP |
Inbound |
Nomad Clients |
API access |
|
443 |
TCP |
Inbound |
Nomad Clients |
API access |
|
80 |
TCP |
Bi-directional |
Services Machine |
Webhooks / API access |
PostgreSQL Servers
Port number | Protocol | Direction | Source / destination | Use | Notes |
---|---|---|---|---|---|
5432 |
TCP |
Bi-directional |
PostgreSQL Servers |
PostgreSQL replication |
Only if using externalized databases. Port is user-defined, assuming the default PostgreSQL port. |
MongoDB Servers
Port number | Protocol | Direction | Source / destination | Use | Notes |
---|---|---|---|---|---|
27017 |
TCP |
Bi-directional |
MongoDB Servers |
MongoDB replication |
Only if using externalized databases. Port is user-defined, assuming the default MongoDB port. |
RabbitMQ Servers
Port number | Protocol | Direction | Source / destination | Use | Notes |
---|---|---|---|---|---|
5672 |
TCP |
Inbound |
Services Machine |
RabbitMQ connection |
Only if using externalized RabbitMQ |
5672 |
TCP |
Bi-directional |
RabbitMQ Servers |
RabbitMQ mirroring |
Only if using externalized RabbitMQ |
Redis Servers
Port number | Protocol | Direction | Source / destination | Use | Notes |
---|---|---|---|---|---|
6379 |
TCP |
Inbound |
Services Machine |
Redis connection |
Only if using externalized Redis |
6379 |
TCP |
Bi-directional |
Redis Servers |
Redis replication |
Only if using externalized Redis, and using Redis replication (optional) |
Nomad Servers
Port number | Protocol | Direction | Source / destination | Use | Notes |
---|---|---|---|---|---|
4646 |
TCP |
Inbound |
Services Machine |
Nomad Server connection |
Only if using externalized Nomad Servers |
4647 |
TCP |
Inbound |
Services Machine |
Nomad Server connection |
Only if using externalized Nomad Servers |
4648 |
TCP |
Bi-directional |
Nomad Servers |
Nomad Servers internal communication |
Only if using externalized Nomad Servers |