Search Results for ""

Installation on AWS with Terraform

Following is a step by step guide to installing CircleCI Server v2.18.2 with Terraform.

Define Variables for Terraform

  1. Clone the Setup repository. If you already have it cloned, make sure it is up-to-date and you are on the master branch by running:

    git checkout master && git pull
  2. Go to the top directory of the enterprise-setup repo on your local machine.

  3. Run terraform init to initialize your working directory.

  4. Run make init to initialize a terraform.tfvars file (your previous terraform.tfvars if any, will be backed up in the same directory).

  5. Open terraform.tfvars in an editor and fill in appropriate AWS values for section 1.

  6. If you plan to use 1.0 builders, specify a circle_secret_passphrase in section 2, replacing …​ with alpha numeric characters, if not, leave it as is. 1.0 builders are disabled by default in section 3.

  7. Specify the instance type to use for your Nomad clients. By default, the value specified in the terraform.tfvars file for Nomad Clients is m4.2xlarge (8 vCPUs, 32GB RAM). To increase the number of concurrent CircleCI jobs that each Nomad Client can run, modify section 2 of the terraform.tfvars file to specify a larger nomad_client_instance_type. Refer to the AWS Amazon EC2 Instance Types guide for details.

    The builder_instance_type is only used for CircleCI 1.0 and is disabled by default in section 3.
  8. In section 3 you can:

    1. choose to use 1.0 Builders if your project requires it (by changing the count to 1)

    2. enter proxy details, and enter a prefix if there will be multiple installations within your AWS region – the Services and Nomad client instances will be displayed with this prefix in the AWS console.

terraform.tfvars
Figure 1. Example tfvars

Above is an example of the terraform.tfvars file you will be editing. The table below shows some of the default settings, and some optional variables that can be used to further customize your cluster. A full list of variables and defaults can be found in the variables.tf file in the root of the enterprise-setup directory.

Optional vars:

Var Description Default

services_instance_type

Instance type for the centralized services box. We recommend a m4 instance

m4.2xlarge

builder_instance_type

Instance type for the 1.0 builder machines. We recommend a r3 instance

r3.2xlarge

max_builders_count

Max number of 1.0 builders

2

nomad_client_instance_type

Instance type for the nomad clients (2.0 builders). We recommend a XYZ instance

m4.xlarge

max_clients_count

Max number of nomad clients

2

prefix

Prefix for resource names

circleci

enable_nomad

Provisions a nomad cluster for CircleCi Server v2.x

1

enable_route

Enable creating a Route53 route for the Services box

0

services_user_data_enabled

Set to 0 to disable automated installation on Services Box

1

force_destroy_s3_bucket

Add/Remove ability to forcefully destroy S3 bucket when your installation is shut down

false

services_disable_api_termination

Protect the services instance from API termination. Set to false if you would like to terminate the Services box automatically when your installation is shut down

true

Provision Instances

  1. Save your changes to the tfvars file and run the following:

    terraform plan
  2. To provision your instances, run the following:

    terraform apply

    You will be asked to confirm if you wish to go ahead by typing yes.

  3. An IP address will be provided at the end of the Terraform output. Visit this IP to carry on the install process.

Access Your Installation

  1. You will see a browser-specific SSL/TLS info box. This is just to inform you that on the next screen your browser might tell you the connection to the admin console is unsafe, but you can be confident it is secure. Click Continue to Setup and proceed to your installation IP.

    SSL Security
    Figure 2. SSL Security
  2. Enter your hostname – this can be your domain name or public IP of the Services Machine instance. At this time you can also upload your SSL public key and certificate if you have them. To proceed without providing these click Use Self-Signed Cert – choosing this option will mean you will see security warnings each time you visit the Management Console.

    Hostname
    Figure 3. Hostname
  3. Upload your license.

  4. Decide how to secure the Management Console. You have three options:

    1. Anonymous admin access to the console, anyone on port 8800 can access (not recommended)

    2. Set a password that can be used to securely access the Management Console (recommended)

    3. Use your existing directory-based authentication system (for example, LDAP)

      Secure the Management Console
      Figure 4. Admin Password
  5. Your CircleCI installation will be put through a set of preflight checks, once they have completed, scroll down and click Continue.

    Preflight Checks
    Figure 5. Preflight Checks

Installation Setup

You should now be on the Management Console settings page (your-circleci-hostname.com:8800).

You can make changes to the settings on this page at any time but changes here will require downtime while the service is restarted. Some settings are covered in more detail in out Operations Guide.
  1. The Hostname field should be pre-populated from earlier in the install process, but if you skipped that step, enter your domain or public IP of the Services machine instance. You can check this has been entered correctly by clicking Test Hostname Resolution.

  2. The Services section is only used when externalizing services. Externalization is available with a Platinum service contract. Contact support@circleci.com if you would like to find out more.

    Hostname and Services Settings
    Figure 6. External Services
  3. Under Execution Engines, only select 1.0 Builders if you require them for a legacy project – most users will leave this unchecked.

  4. Select Cluster in the 2.0 Builders Configuration section. The Single box option will run jobs on the Services machine, rather than a dedicated instance, so is only suitable for trialling the system, or for some small teams.

    Execution Engine
    Figure 7. 1.0 and 2.0 Builders
  5. Register CircleCI as a new OAuth application in GitHub.com or GitHub Enterprise by following the instructions provided onscreen.

    If you get an "Unknown error authenticating via GitHub. Try again, or contact us." message, try using http: instead of https: for the Homepage URL and callback URL.
  6. Copy the Client ID and Secret from GitHub and paste it into the relevant fields, then click Test Authentication.

  7. If you are using GitHub.com, move on to the next step. If using Github Enterprise, you will also need to supply an API Token so we can verify your organization. To provide this, complete the following from your GitHub Enterprise dashboard:

    1. Navigate to Personal Settings (top right) > Developer Settings > Personal Access Tokens.

    2. Click “generate new token”. Name the token appropriately to prevent accidental deletion. Do not tick any of the checkboxes, we only require the default public read-level access so no extra permissions are required. We recommend this token should be shared across your organization rather than being owned by a single user.

    3. Copy the new token and paste it into the GitHub Enterprise Default API Token field.

      Github Integration
      Figure 8. Enter Github Enterprise Token
  8. If you wish to use LDAP authentication for your installation, enter the required details in the LDAP section.

  9. We recommend using an SSL certificate and key for your install. You can submit these in the Privacy section if this step was missed during the installation.

    privacy
    Figure 9. Privacy Settings
  10. We recommend using S3 for storage and all required fields for Storage are pre-populated. The IAM user, as referred to in the planning section of this document, is used here.

    storage
    Figure 10. Storage Options
  11. Complete enhanced AWS Integration options.

  12. Complete the Email section if you wish to configure your own email server for sending build update emails. Leave this section is you wish to use our default email server.

    Due to an issue with our third party tooling, Replicated, the Test SMTP Authentication button is not currently working
  13. Configure VM service if you plan to use Remote Docker or machine executor features. We recommend using an IAM instance profile for authentication, as described in the planning section of this document. With this section completed, instances will automatically be provisioned to execute jobs in Remote Docker or use the machine executor. For more information on VM Service and creating custom AMIs for remote Docker and machine executor jobs, see our VM service guide.

    vmprovider
    Figure 11. Configure VM Service

    You can preallocate instances to always be up and running, reducing the time taken for Remote Docker and machine executor jobs to start. If preallocation is set, a cron job will cycle through your preallocated instances once per day to prevent them getting into a bad/dead state.

    If Docker Layer Caching (DLC) is to be used VM preallocation must be set to 0 – on-demand – for both Remote Docker and machine executor.
  14. If you wish to use AWS Cloudwatch or Datadog for collating metrics for your installation, set this up here. For more information see our Monitoring guidance:

    metrics setup
    Figure 12. Metrics

    You can also customize the metrics received through Telegraf. For more on this see our Custom Metics guide.

  15. Artifacts persist data after a job is completed, and may be used for longer-term storage of your build process outputs. By default, CircleCI Server only allows approved types to be served. This is to protect users from uploading, and potentially executing malicious content. The Artifacts setting allows you to override this protection. For more information on safe/unsafe types see our Build Artifacts guidance.

  16. After agreeing to the License Agreement and saving your settings, select Restart Now from the popup. You will then be redirected to start CircleCI and view the Management Console Dashboard. It will take a few minutes to download all of the necessary Docker containers.

If the Management Console reports Failure reported from operator: no such image click Start again and it should continue.

Validate Your Installation

  1. When the application is started, select Open to launch CircleCI in your browser, and sign up/log in to your CircleCI installation and start running 2.0 builds! You will become the Administrator at this point as you are the first person to sign in. Have a look at our Getting Started guide to start adding projects.

    dashboard
    Figure 13. Start CircleCI from your Dashboard
  2. After build containers have started and images have been downloaded, the first build should begin immediately. If there are no updates after around 15 minutes, and you have clicked the Refresh button, contact CircleCI support for assistance.

  3. Next, use our realitycheck repo to check basic CircleCI functionality.

  4. If you’re unable to run your first builds successfully please start with our Troubleshooting guide for general troubleshooting topics, and our Introduction to Nomad Cluster Operation for information about how to check the status of Builders in your installation.