Installation on AWS with Terraform

Following is a step by step guide to installing CircleCI Server v2.19.8 with Terraform.

Define Variables for Terraform

  1. Clone the Setup repository. If you already have it cloned, make sure it is up-to-date and you are on the master branch by running:

    git checkout master && git pull
  2. Go to the top directory of the enterprise-setup repo on your local machine.

  3. Run terraform init to initialize your working directory.

  4. Run make init to initialize a terraform.tfvars file (your previous terraform.tfvars if any, will be backed up in the same directory).

  5. Open terraform.tfvars in an editor and fill in appropriate AWS values for section 1.

  6. If you plan to use 1.0 builders, specify a circle_secret_passphrase in section 2, replacing …​ with alpha numeric characters, if not, leave it as is. 1.0 builders are disabled by default in section 3.

  7. Specify the instance type to use for your Nomad clients. By default, the value specified in the terraform.tfvars file for Nomad Clients is m5.2xlarge (8 vCPUs, 32GB RAM). To increase the number of concurrent CircleCI jobs that each Nomad Client can run, modify section 2 of the terraform.tfvars file to specify a larger nomad_client_instance_type. Refer to the AWS Amazon EC2 Instance Types guide for details, and see our guidance in the System Requirements document.

    The builder_instance_type is only used for CircleCI 1.0 and is disabled by default in section 3.
  8. In section 3 you can:

    1. choose to use 1.0 Builders if your project requires it (by changing the count to 1)

    2. enter proxy details, and enter a prefix if there will be multiple installations within your AWS region – the Services and Nomad client instances will be displayed with this prefix in the AWS console.

Figure 1. Example tfvars

Above is an example of the terraform.tfvars file you will be editing. The table below shows some of the default settings, and some optional variables that can be used to further customize your cluster. A full list of variables and defaults can be found in the file in the root of the enterprise-setup directory.

If you require your installation to work on AWS GovCloud, you can enable this by setting enable_govcloud to true.

Optional vars:

Var Description Default


Instance type for the centralized services box. We recommend a m4 instance



Instance type for the 1.0 builder machines. We recommend a r3 instance



Max number of 1.0 builders



Instance type for the nomad clients (2.0 builders). We recommend a XYZ instance



Max number of nomad clients



Prefix for resource names



Provisions a nomad cluster for CircleCi Server v2.x



Enable creating a Route53 route for the Services box



Allows deployment into AWS GovCloud



Set to 0 to disable automated installation on Services Box



Add/Remove ability to forcefully destroy S3 bucket when your installation is shut down



Protect the services instance from API termination. Set to false if you would like to terminate the Services box automatically when your installation is shut down


Provision Instances

  1. Save your changes to the tfvars file and run the following:

    terraform plan
  2. To provision your instances, run the following:

    terraform apply

    You will be asked to confirm if you wish to go ahead by typing yes.

  3. An IP address will be provided at the end of the Terraform output. Visit this IP to carry on the install process.

Access Your Installation

  1. You will see a browser-specific SSL/TLS info box. This is just to inform you that on the next screen your browser might tell you the connection to the admin console is unsafe, but you can be confident it is secure. Click Continue to Setup and proceed to your installation IP.

    SSL Security
    Figure 2. SSL Security
  2. Enter your hostname. This can be your domain name or public IP of the Services Machine instance. At this time you can also upload your SSL public key and certificate if you have them. To proceed without providing these click Use Self-Signed Cert – choosing this option will mean you will see security warnings each time you visit the Management Console.

    Figure 3. Hostname
  3. Upload your license.

  4. Decide how to secure the Management Console. You have three options:

    1. Anonymous admin access to the console, anyone on port 8800 can access (not recommended)

    2. Set a password that can be used to securely access the Management Console (recommended)

    3. Use your existing directory-based authentication system (for example, LDAP)

      Secure the Management Console
      Figure 4. Admin Password
  5. Your CircleCI installation will be put through a set of preflight checks, once they have completed, scroll down and click Continue.

    Preflight Checks
    Figure 5. Preflight Checks

Installation Setup

You should now be on the Management Console settings page (

You can make changes to the settings on this page at any time but changes here will require downtime while the service is restarted. Some settings are covered in more detail in out Operations Guide.
  1. Hostname – The Hostname field should be pre-populated from earlier in the install process, but if you skipped that step, enter your domain or public IP of the Services machine instance. You can check this has been entered correctly by clicking Test Hostname Resolution.

  2. Services – The Services section is only used when externalizing services. Externalization is available with a Platinum service contract. Contact if you would like to find out more.

    Hostname and Services Settings
    Figure 6. External Services
  3. Execution Engines – only select 1.0 Builders if you require them for a legacy project – most users will leave this unchecked.

  4. Builders Configuration – select Cluster in the 2.0 section. The Single box option will run jobs on the Services machine, rather than a dedicated instance, so is only suitable for trialling the system, or for some small teams.

    Execution Engine
    Figure 7. 1.0 and 2.0 Builders
  5. GitHub Integration – register CircleCI as a new OAuth application in or GitHub Enterprise by following the instructions provided on the page.

    If you get an "Unknown error authenticating via GitHub. Try again, or contact us." message, try using http: instead of https: for the Homepage URL and callback URL.
    1. Copy the Client ID and Secret from GitHub and paste it into the relevant fields, then click Test Authentication.

    2. If you are using, move on to step 6. If using Github Enterprise, you will also need to follow some suplementary steps and supply an API Token so we can verify your organization. To provide this, complete the following from your GitHub Enterprise dashboard:

      1. Navigate to Personal Settings (top right) > Developer Settings > Personal Access Tokens.

      2. Click “generate new token”. Name the token appropriately to prevent accidental deletion. Do not tick any of the checkboxes, we only require the default public read-level access so no extra permissions are required. We recommend this token should be shared across your organization rather than being owned by a single user.

      3. Copy the new token and paste it into the GitHub Enterprise Default API Token field.

        Github Integration
        Figure 8. Enter Github Enterprise Token
  6. LDAP – if you wish to use LDAP authentication for your installation, enter the required details in the LDAP section. For a detailed runthrough of LDAP settings, see our LDAP authentication guide

  7. Privacy – We recommend using an SSL certificate and key for your install. You can submit these in the Privacy section if this step was missed during the installation.

    Figure 9. Privacy Settings
  8. Storage – We recommend using S3 for storage and all required fields for Storage are pre-populated. The IAM user, as referred to in the planning section of this document, is used here.

    Figure 10. Storage Options
  9. Enhanced AWS Integration – Complete this section if you are using 1.0 builders.

  10. Email Complete the Email section if you wish to configure your own email server for sending build update emails. Leave this section is you wish to use our default email server.

    Due to an issue with our third party tooling, Replicated, the Test SMTP Authentication button is not currently working
  11. VM Provider – Configure VM service if you plan to use Remote Docker or machine executor (Linux/Windows) features. We recommend using an IAM instance profile for authentication, as described in the planning section of this document. With this section completed, instances will automatically be provisioned to execute jobs in Remote Docker or use the machine executor. To use the Windows machine executor you will need to build an image. For more information on VM Service and creating custom AMIs for remote Docker and machine executor jobs, see our VM service guide.

    You can preallocate instances to always be up and running, reducing the time taken for Remote Docker and machine executor jobs to start. If preallocation is set, a cron job will cycle through your preallocated instances once per day to prevent them getting into a bad/dead state.

    If Docker Layer Caching (DLC) is to be used, VM preallocation should be set to 0, forcing containers to be spun up on-demand for both machine and Remote Docker. It is worth noting here that if these fields are not set to 0 but all preallocated instances are in use, DLC will work correctly, as if preallocation was set to 0.
  12. AWS Cloudwatch or Datadog Metrics can be configured for your installation. Set either of these up in the relevant sections. For more information see our Monitoring guidance:

    metrics setup
    Figure 11. Metrics
  13. Custom Metrics are an alternative to Cloudwatch and Datadog metrics, you can also customize the metrics you receive through Telegraf. For more on this see our Custom Metics guide.

  14. Distributed Tracing is used in our support bundles, and settings should remain set to default unless a change is requested by CircleCI Support.

  15. Artifacts persist data after a job is completed, and may be used for longer-term storage of your build process outputs. By default, CircleCI Server only allows approved types to be served. This is to protect users from uploading, and potentially executing malicious content. The Artifacts setting allows you to override this protection. For more information on safe/unsafe types see our Build Artifacts guidance.

  16. After agreeing to the License Agreement and saving your settings, select Restart Now from the popup. You will then be redirected to start CircleCI and view the Management Console Dashboard. It will take a few minutes to download all of the necessary Docker containers.

If the Management Console reports Failure reported from operator: no such image click Start again and it should continue.

Validate Your Installation

  1. When the application is started, select Open to launch CircleCI in your browser, and sign up/log in to your CircleCI installation and start running 2.0 builds! You will become the Administrator at this point as you are the first person to sign in. Have a look at our Getting Started guide to start adding projects.

    Figure 12. Start CircleCI from your Dashboard
  2. After build containers have started and images have been downloaded, the first build should begin immediately. If there are no updates after around 15 minutes, and you have clicked the Refresh button, contact CircleCI support for assistance.

  3. Next, use our realitycheck repo to check basic CircleCI functionality.

  4. If you’re unable to run your first builds successfully please start with our Troubleshooting guide for general troubleshooting topics, and our Introduction to Nomad Cluster Operation for information about how to check the status of Builders in your installation.