Using the Static Installation Scripts

This article provides a System Administrators’ overview of CircleCI’s 2.0 static installation for non-AWS environments.

• Limitations
• Build Environments
• Architecture
  • Services
  • Nomad Clients
  • GitHub
• Installation
  • Prerequisites
  • Installing the Services Machine
  • Installing the Nomad Clients
  • Storage
  • Troubleshooting

Limitations

This method of installation has the following limitations:

• It is not possible to use machine executors.
• It is not possible to use the Remote Docker Environment or Docker Layer Caching.
• There is no first-class high-availability option.

CircleCI 2.0 provides new infrastructure that includes the following improvements:

• New configuration with any number of jobs and workflows to orchestrate them.
• Custom images for execution on a per-job basis.
• Fine-grained performance with custom caching and per-job CPU or memory allocation.

Build Environments

By default, CircleCI 2.0’s Nomad Client instances automatically provision containers according to the image configured for each job in your .circleci/config.yml file. CircleCI uses Nomad as the primary job scheduler in CircleCI 2.0. Refer to the Introduction to Nomad Cluster Operation to learn more about the job scheduler and how to perfom basic client and cluster operations.

Architecture

A CircleCI static installation consists of two primary components: Services and Nomad Clients. Services run on a single instance that is comprised of the core application, storage, and networking functionality. Any number of Nomad Clients execute jobs and communicate back to the Services machine. Both components must access an instance of GitHub or GitHub Enterprise on the network as illustrated in the following architecture diagram.

Services

The machine on which the Services instance runs should only be restarted gracefully and may be backed up using built-in VM snapshotting. Note: It is possible to configure external data storage with PostgreSQL and Mongo for high availability and then use standard tooling for database backups, see Adding External Database Hosts for High Availability. DNS resolution must point to the IP address of the machine on which the Services are installed. The following table describes the ports used for traffic on the Service instance:

Nomad Clients

The Nomad Client instances run without storing state, enabling you to increase or decrease containers as needed. To ensure that there are enough client machines running to handle all of the builds, track the queued builds, and increase the client machines as needed to balance the load.

Each machine on which the Nomad Clients are installed reserves two CPUs and 4GB of memory for coordinating builds. The remaining processors and memory create the containers. Larger machines are able to run more containers and are limited by the number of available cores after two are reserved for coordination. The following table describes the ports used on the Nomad client instances:

GitHub

CircleCI uses GitHub or GitHub Enterprise credentials for authentication which, in turn, may use LDAP, SAML, or SSH for access. CircleCI will inherit the authentication supported by your central SSO infrastructure. The following table describes the ports used on machines running GitHub to communicate with the Services and Nomad client instances.

Installation

The following sections describe the steps for installation of the Services VM and the Nomad cluster.

Prerequisites

Have the following available before beginning the installation procedure:

• A CircleCI License file (.rli). Contact CircleCI support if you need a license.
• A machine to run Ubuntu 14.04 or 16.04 with a minimum of at least 100 GB storage, 32 GB RAM, and 4 CPUs (8 CPUs preferred) for the Services VM.
• A cluster of machines running Ubuntu 14.04 or 16.04 with a minumum of 8 GB RAM and 4 CPUs each, as well as network access to any Docker registries that are required by your builds for the Nomad Client VMs.

Installing the Services Machine

1. Copy the Services init script to the Services VM machine.

2. Log in to the machine provisioned for the Services VM and run the sudo su command.

3. Run ./provision-services-ubuntu.sh to start the script.

4. Go to the public IP of the host on port 8800 using HTTPS.

5. Enter your license.

6. Complete the Storage section. If you are not using a cloud service, then you will pick None (more information below).

7. Set the VM Provider to None.

8. Set 1.0 Builds to Off.

9. Set 2.0 Builds to Clustered.

Installing the Nomad Clients

1. Copy the Client init script to the Nomad Server machine.

2. Log in to the machine provisioned for the Nomad Server and run the sudo su command.

3. To start the script, run ./provision-nomad-client-ubuntu.sh with the NOMAD_SERVER_ADDRESS environment variable set to the routable IP of the Services machine (for example, NOMAD_SERVER_ADDRESS=0.0.0.0 ./provision-nomad-client-ubuntu.sh).

Storage

The None storage driver saves all of your CircleCI data locally. This means that artifacts, test results, and action logs will be saved locally at /data/circle/storage-fileserver. It is best practice to mount an external volume and create a symbolic link between the two when using this storage option. Note: Data may only be transferred as quickly as the external volume will allow, so SSDs are best practice.

Troubleshooting

This section includes some possible resolutions for common issues that may be encountered during system setup and installation.

• Symptom: Jobs stay in queued status until they fail and never successfully run.
  • Check port 8585 if the nomad client logs contain the following type of error message:
    • {“error”:”rpc error: code = Unavailable desc = grpc: the connection is unavailable”,”level”:”warning”,”msg”:”error fetching config, retrying”,”time”:”2018-04-17T18:47:01Z”}