System Requirements

This section defines the system and port access requirements for installing CircleCI v2.19.11.

Services Machine
Nomad Clients
- Recommendations for Heavy Usage
Choosing Nomad Client Quantity
Server Ports

Services Machine

The Services machine hosts the core of our Server product, including the user-facing website, API engine, datastores, and Nomad job scheduler. It is best practice to use an isolated machine.

Currently, CircleCI Server only supports x86_64 architecture.

The following table defines the Services machine CPU, RAM, and disk space requirements:

Number of daily active CircleCI users	CPU	RAM	Disk space	NIC speed
<50	8 cores	32GB	100GB	1Gbps
50-250	12 cores	64GB	200GB	1Gbps
251-1000	16 cores	128GB	500GB	10Gbps
1001-5000	20 cores	256GB	1TB	10Gbps
5000+	24 cores	512GB	2TB	10Gbps

Nomad Clients

Nomad client machines run CircleCI jobs that are scheduled by the Nomad Server. The Minimum CPU, RAM, and disk space requirements per client are as follows:

CPU: 4 cores
RAM: 32GB
Disk space: 100GB
NIC speed: 1Gbps

For an AWS install of CircleCI Server, the recommended instance type for Nomad clients is m5.2xlarge (8 vCPUs @ 2.4GHz, 32GB RAM).

Currently, Nomad Clients only support x86_64 architecture.

You can choose a larger instance type to fit more jobs per Client. To help in this choice, consider that when Nomad decides if a job will fit on a Client, the Job is allocated 1024MHz per CPU, and capacity is number of cores * clock speed. Using this method, the maximum capacity of a m5.2xlarge would be 19200MHz, which would mean 9.6 jobs could run on that client (if there were no limiting factors). In practice, Nomad will researve some CPU for itself, and because of the CPU:RAM ratio, the available RAM is the limiting factor governing how many jobs can run.

Recommendations for Heavy Usage

EBS will throttle IOPS after certain IO operations, therefore heavy-load usage with the default Server configuration (m5.2xlarge with 200-GiB gp2 root volume) can see performance-related issues, such as slow jobs, slow spin-up, and job timeouts in worst cases.

To give an idea of what is meant by heavy usage:

If BurstBalance metrics for storage attached to Nomad clients are decreasing sharply, this is a sign that the workload is heavy enough to be affected. See the AWS EBS metrics document for further details.
We have had a report from a customer that continuous disk I/O at 500 MB/s used up the burst credits for our default storage configuration within 37.5 minutes.

Suggested workarounds for this issue are as follows:

Use EBS volumes larger than 1 TiB. Large EBS volumes never consume IO credits.
Use the *5d instance type and use an attached instance store for Docker-related files (i.e. /var/lib/docker).

Choosing Nomad Client Quantity

The following table defines the number of Nomad clients to make available as a best practice. Scale up and down according to demand on your system:

Number of daily active CircleCI users	Number of Nomad client machines
<50	1-5
50-250	5-10
250-1000	10-15
5000+	15+

Server Ports

Below all ports required by a CircleCI 2.0 installation are listed for each machine type.

Services Machine

Port number	Protocol	Direction	Source / destination	Use	Notes
80	TCP	Inbound	End users	HTTP web app traffic
443	TCP	Inbound	End users	HTTPS web app traffic
7171	TCP	Inbound	End users	Artifacts access
8081	TCP	Inbound	End users	Artifacts access
22	TCP	Inbound	Administrators	SSH
8800	TCP	Inbound	Administrators	Admin console
8125	UDP	Inbound	Nomad Clients	Metrics
8125	UDP	Inbound	Nomad Servers	Metrics	Only if using externalized Nomad Servers
8125	UDP	Inbound	All Database Servers	Metrics	Only if using externalised databases
4647	TCP	Bi-directional	Nomad Clients	Internal communication
8585	TCP	Bi-directional	Nomad Clients	Internal communication
7171	TCP	Bi-directional	Nomad Clients	Internal communication
3001	TCP	Bi-directional	Nomad Clients	Internal communication
80	TCP	Bi-directional	GitHub Enterprise / GitHub.com (whichever applies)	Webhooks / API access
443	TCP	Bi-directional	GitHub Enterprise / GitHub.com (whichever applies)	Webhooks / API access
80	TCP	Outbound	AWS API endpoints	API access	Only if running on AWS
443	TCP	Outbound	AWS API endpoints	API access	Only if running on AWS
5432	TCP	Outbound	PostgreSQL Servers	PostgreSQL database connection	Only if using externalised databases. Port is user-defined, assuming the default PostgreSQL port.
27017	TCP	Outbound	MongoDB Servers	MongoDB database connection	Only if using externalized databases. Port is user-defined, assuming the default MongoDB port.
5672	TCP	Outbound	RabbitMQ Servers	RabbitMQ connection	Only if using externalized RabbitMQ
6379	TCP	Outbound	Redis Servers	Redis connection	Only if using externalized Redis
4647	TCP	Outbound	Nomad Servers	Nomad Server connection	Only if using externalized Nomad Servers
443	TCP	Outbound	CloudWatch Endpoints	Metrics	Only if using AWS CloudWatch

Nomad Clients

Port number	Protocol	Direction	Source / destination	Use	Notes
64535-65535	TCP	Inbound	End users	SSH into builds feature
80	TCP	Inbound	Administrators	CircleCI Admin API access
443	TCP	Inbound	Administrators	CircleCI Admin API access
22	TCP	Inbound	Administrators	SSH
22	TCP	Outbound	GitHub Enterprise / GitHub.com (whichever applies)	Download Code From GitHub.
4647	TCP	Bi-directional	Services Machine	Internal communication
8585	TCP	Bi-directional	Services Machine	Internal communication
7171	TCP	Bi-directional	Services Machine	Internal communication
3001	TCP	Bi-directional	Services Machine	Internal communication
443	TCP	Outbound	Cloud Storage Provider	Artifacts storage	Only if using external artifacts storage
53	UDP	Outbound	Internal DNS Server	DNS resolution	This is to make sure that your jobs can resolve all DNS names that are needed for their correct operation.

GitHub Enterprise / GitHub.com

Port number	Protocol	Direction	Source / destination	Use
22	TCP	Inbound	Services Machine	Git access
22	TCP	Inbound	Nomad Clients	Git access
80	TCP	Inbound	Nomad Clients	API access
443	TCP	Inbound	Nomad Clients	API access
80	TCP	Bi-directional	Services Machine	Webhooks / API access

PostgreSQL Servers

Port number	Protocol	Direction	Source / destination	Use	Notes
5432	TCP	Bi-directional	PostgreSQL Servers	PostgreSQL replication	Only if using externalized databases. Port is user-defined, assuming the default PostgreSQL port.

MongoDB Servers

Port number	Protocol	Direction	Source / destination	Use	Notes
27017	TCP	Bi-directional	MongoDB Servers	MongoDB replication	Only if using externalized databases. Port is user-defined, assuming the default MongoDB port.

RabbitMQ Servers

Port number	Protocol	Direction	Source / destination	Use	Notes
5672	TCP	Inbound	Services Machine	RabbitMQ connection	Only if using externalized RabbitMQ
5672	TCP	Bi-directional	RabbitMQ Servers	RabbitMQ mirroring	Only if using externalized RabbitMQ

Redis Servers

Port number	Protocol	Direction	Source / destination	Use	Notes
6379	TCP	Inbound	Services Machine	Redis connection	Only if using externalized Redis
6379	TCP	Bi-directional	Redis Servers	Redis replication	Only if using externalized Redis, and using Redis replication (optional)

Nomad Servers

Port number	Protocol	Direction	Source / destination	Use	Notes
4646	TCP	Inbound	Services Machine	Nomad Server connection	Only if using externalized Nomad Servers
4647	TCP	Inbound	Services Machine	Nomad Server connection	Only if using externalized Nomad Servers
4648	TCP	Bi-directional	Nomad Servers	Nomad Servers internal communication	Only if using externalized Nomad Servers