Search Results for ""

System Requirements

This document is intended for system administrators of self-hosted installations of CircleCI Server.

This section defines the system and port access requirements for installing CircleCI v2.19.5.

Services Machine

The Services machine hosts the core of our Server product, including the user-facing website, API engine, datastores, and Nomad job scheduler. It is best practice to use an isolated machine.

The following table defines the Services machine CPU, RAM, and disk space requirements:

Number of daily active CircleCI users CPU RAM Disk space NIC speed

<50

8 cores

32GB

100GB

1Gbps

50-250

12 cores

64GB

200GB

1Gbps

251-1000

16 cores

128GB

500GB

10Gbps

1001-5000

20 cores

256GB

1TB

10Gbps

5000+

24 cores

512GB

2TB

10Gbps

Nomad Clients

Nomad client machines run CircleCI jobs that are scheduled by the Nomad Server. The Minimum CPU, RAM, and disk space requirements per client are as follows:

  • CPU: 4 cores

  • RAM: 32GB

  • Disk space: 100GB

  • NIC speed: 1Gbps

For an AWS install of CircleCI Server, the recommended instance type for Nomad clients is m5.2xlarge (8 vCPUs @ 2.4GHz, 32GB RAM).

You can choose a larger instance type to fit more jobs per Client. To help in this choice, consider that when Nomad decides if a job will fit on a Client, the Job is allocated 1024MHz per CPU, and capacity is number of cores * clock speed. Using this method, the maximum capacity of a m5.2xlarge would be 19200MHz, which would mean 9.6 jobs could run on that client (if there were no limiting factors). In practice, Nomad will researve some CPU for itself, and because of the CPU:RAM ratio, the available RAM is the limiting factor governing how many jobs can run.

Recommendations for Heavy Usage

EBS will throttle IOPS after certain IO operations, therefore heavy-load usage with the default Server configuration (m5.2xlarge with 200-GiB gp2 root volume) can see performance-related issues, such as slow jobs, slow spin-up, and job timeouts in worst cases.

To give an idea of what is meant by heavy usage:

  • If BurstBalance metrics for storage attached to Nomad clients are decreasing sharply, this is a sign that the workload is heavy enough to be affected. See the AWS EBS metrics document for further details.

  • We have had a report from a customer that continuous disk I/O at 500 MB/s used up the burst credits for our default storage configuration within 37.5 minutes.

Suggested workarounds for this issue are as follows:

  • Use EBS volumes larger than 1 TiB. Large EBS volumes never consume IO credits.

  • Use the *5d instance type and use an attached instance store for Docker-related files (i.e. /var/lib/docker).

Choosing Nomad Client Quantity

The following table defines the number of Nomad clients to make available as a best practice. Scale up and down according to demand on your system:

Number of daily active CircleCI users Number of Nomad client machines

<50

1-5

50-250

5-10

250-1000

10-15

5000+

15+

Server Ports

Below all ports required by a CircleCI 2.0 installation are listed for each machine type.

Services Machine

Port number Protocol Direction Source / destination Use Notes

80

TCP

Inbound

End users

HTTP web app traffic

443

TCP

Inbound

End users

HTTPS web app traffic

7171

TCP

Inbound

End users

Artifacts access

8081

TCP

Inbound

End users

Artifacts access

22

TCP

Inbound

Administrators

SSH

8800

TCP

Inbound

Administrators

Admin console

8125

UDP

Inbound

Nomad Clients

Metrics

8125

UDP

Inbound

Nomad Servers

Metrics

Only if using externalized Nomad Servers

8125

UDP

Inbound

All Database Servers

Metrics

Only if using externalised databases

4647

TCP

Bi-directional

Nomad Clients

Internal communication

8585

TCP

Bi-directional

Nomad Clients

Internal communication

7171

TCP

Bi-directional

Nomad Clients

Internal communication

3001

TCP

Bi-directional

Nomad Clients

Internal communication

80

TCP

Bi-directional

GitHub Enterprise / GitHub.com (whichever applies)

Webhooks / API access

443

TCP

Bi-directional

GitHub Enterprise / GitHub.com (whichever applies)

Webhooks / API access

80

TCP

Outbound

AWS API endpoints

API access

Only if running on AWS

443

TCP

Outbound

AWS API endpoints

API access

Only if running on AWS

5432

TCP

Outbound

PostgreSQL Servers

PostgreSQL database connection

Only if using externalised databases. Port is user-defined, assuming the default PostgreSQL port.

27017

TCP

Outbound

MongoDB Servers

MongoDB database connection

Only if using externalized databases. Port is user-defined, assuming the default MongoDB port.

5672

TCP

Outbound

RabbitMQ Servers

RabbitMQ connection

Only if using externalized RabbitMQ

6379

TCP

Outbound

Redis Servers

Redis connection

Only if using externalized Redis

4647

TCP

Outbound

Nomad Servers

Nomad Server connection

Only if using externalized Nomad Servers

443

TCP

Outbound

CloudWatch Endpoints

Metrics

Only if using AWS CloudWatch

Nomad Clients

Port number Protocol Direction Source / destination Use Notes

64535-65535

TCP

Inbound

End users

SSH into builds feature

80

TCP

Inbound

Administrators

CircleCI Admin API access

443

TCP

Inbound

Administrators

CircleCI Admin API access

22

TCP

Inbound

Administrators

SSH

22

TCP

Outbound

GitHub Enterprise / GitHub.com (whichever applies)

Download Code From GitHub.

4647

TCP

Bi-directional

Services Machine

Internal communication

8585

TCP

Bi-directional

Services Machine

Internal communication

7171

TCP

Bi-directional

Services Machine

Internal communication

3001

TCP

Bi-directional

Services Machine

Internal communication

443

TCP

Outbound

Cloud Storage Provider

Artifacts storage

Only if using external artifacts storage

53

UDP

Outbound

Internal DNS Server

DNS resolution

This is to make sure that your jobs can resolve all DNS names that are needed for their correct operation.

GitHub Enterprise / GitHub.com

Port number Protocol Direction Source / destination Use Notes

22

TCP

Inbound

Services Machine

Git access

22

TCP

Inbound

Nomad Clients

Git access

80

TCP

Inbound

Nomad Clients

API access

443

TCP

Inbound

Nomad Clients

API access

80

TCP

Bi-directional

Services Machine

Webhooks / API access

PostgreSQL Servers

Port number Protocol Direction Source / destination Use Notes

5432

TCP

Bi-directional

PostgreSQL Servers

PostgreSQL replication

Only if using externalized databases. Port is user-defined, assuming the default PostgreSQL port.

MongoDB Servers

Port number Protocol Direction Source / destination Use Notes

27017

TCP

Bi-directional

MongoDB Servers

MongoDB replication

Only if using externalized databases. Port is user-defined, assuming the default MongoDB port.

RabbitMQ Servers

Port number Protocol Direction Source / destination Use Notes

5672

TCP

Inbound

Services Machine

RabbitMQ connection

Only if using externalized RabbitMQ

5672

TCP

Bi-directional

RabbitMQ Servers

RabbitMQ mirroring

Only if using externalized RabbitMQ

Redis Servers

Port number Protocol Direction Source / destination Use Notes

6379

TCP

Inbound

Services Machine

Redis connection

Only if using externalized Redis

6379

TCP

Bi-directional

Redis Servers

Redis replication

Only if using externalized Redis, and using Redis replication (optional)

Nomad Servers

Port number Protocol Direction Source / destination Use Notes

4646

TCP

Inbound

Services Machine

Nomad Server connection

Only if using externalized Nomad Servers

4647

TCP

Inbound

Services Machine

Nomad Server connection

Only if using externalized Nomad Servers

4648

TCP

Bi-directional

Nomad Servers

Nomad Servers internal communication

Only if using externalized Nomad Servers