Upgrading a Server Installation

This document describes the process for upgrading your CircleCI Server installation to v2.19.

Nomad Launch Configuration

Before upgrading to v2.19, follow this guide to update your nomad launch configuration.

Org Rename Script

If upgrading directly from v2.17 (or below) and you have ever had issues with renaming an organization within CircleCI or you suspect that an organization rename might have happened at any point follow the steps below. If you are running Server v2.18.x already you can skip this section.
  1. SSH into your Services machine

  2. REPL into workflows-conductor by running the following: sudo docker exec -it workflows-conductor lein repl :connect 6005

  3. Go to this link for the org rename script. Copy/paste this script into the REPL session. It will run migration and output current progress.

  4. If any ERROR messages are present in the output please report back to your CSM or reach out to support.

Upgrade Steps Overview

Follow these steps to upgrade your CircleCI Server version.

1. Snapshot for Rollback

This step is optional but recommended

To take a snapshot of your installation:

  1. Go to the Management Console (e.g. your-circleci-hostname.com:8800) and click Stop Now to stop the CircleCI service.

    Stop CircleCI
    Figure 1. Stop CircleCI
  2. Ensure no jobs are running on the nomad clients – you can check this by running nomad status

  3. Navigate to the AWS EC2 management console and select your Services machine instance

  4. Select Actions > Image > Create Image – Select the No Reboot option if you want to avoid downtime at this point. This image creation step creates an AMI that can be readily launched as a new EC2 instance to restore your installation.

    Snapshot Image Creation
    Figure 2. Snapshot Image Creation
It is also possible to automate this process with the AWS API. Subsequent AMIs/snapshots are only as large as the difference (changed blocks) since the last snapshot, such that storage costs are not necessarily larger for more frequent snapshots, see Amazon’s EBS snapshot billing document for details. Once you have the snapshot you are free to make changes on the Services machine.

If you do need to rollback at any point, see our guide to restoring from a backup.

2. Updating Replicated

a. Prerequisites

  • Your installation is Ubuntu 14.04 or 16.04 based.

  • Your installation is not airgapped and you can access the internet from it.

  • We will be updating to Replicated v2.38.6, but first we need to check you are running at least v2.10.3 on your Services machine. To check this, SSH into the Services machine and run the following:

    replicated --version

    If you are running a version of Replicated pre v2.10.3 please reach out to support@circleci.com. If you are already on v2.38.6 or above you can skip the next step and move to upgrade the CircleCI application

b. Preparations

Remember to take a snapshot (described above) before starting the Replicated update process
  1. Stop the CircleCI application by clicking the Stop Now button on the Dashboard. Application shutdown takes a few minutes. Wait for the status to become “Stopped” before continuing.

    Stop the CircleCI Application
    Figure 3. Stop the CircleCI Application

    Alternatively you can SSH into the Services machine and stop the CircleCI application from the command line:

    replicatedctl app stop

    You can check the status using the followng:

    replicatedctl app status inspect

    Example Output:

            "AppID": "edd9471be0bc4ea04dfca94718ddf621",
            "Sequence": 2439,
            "State": "stopped",
            "DesiredState": "stopped",
            "Error": "",
            "IsCancellable": false,
            "IsTransitioning": false,
            "LastModifiedAt": "2018-10-23T22:00:21.314987894Z"
  2. For the replicated update to succeed, it is necessary to update docker to the recommended version, 17.12.1. Check which version you are running with docker version and if you need to update, follow these steps:

    sudo apt-get install docker-ce=17.12.1~ce-0~ubuntu
  3. Pin the Docker version using the following command:

    sudo apt-mark hold docker-ce

c. Perform Update

  1. Perform the Replicated update by executing the update script as follows:

    curl -sSL "https://get.replicated.com/docker?replicated_tag=2.38.6" | sudo bash

    Double-check your replicated and docker versions:

    replicatedctl version    # 2.38.6
    docker -v                # 17.12.1
  2. Restart the app with

    replicatedctl app start

    The application will take a few minutes to spin up. You can check the progress in the administration dashboard or by executing;

    replicatedctl app status inspect

    Example output:

            "AppID": "edd9471be0bc4ea04dfca94718ddf621",
            "Sequence": 2439,
            "State": "started",
            "DesiredState": "started",
            "Error": "",
            "IsCancellable": true,
            "IsTransitioning": true,
            "LastModifiedAt": "2018-10-23T22:04:05.00374451Z"

3. Upgrade CircleCI Server

  1. Once you are running the latest version of Replicated, click the View Update button in the Management Console dashboard.

    View Available Updates
    Figure 4. View Available Updates
  2. Click Install next to the version you wish to install.

    Please refresh your screen intermittently during the install process to avoid unnecessary waiting.
    View Available Releases
    Figure 5. View Available Releases

    The install process may take several minutes and the install status will be displayed both on the Releases page and the main Dashboard.

  3. Once the installation is finished, navigate to the Dashboard to start your installation - Note the middle box on the Dashboard will read "CircleCI is up to date" when you are running the latest version.

Help make this document better

This guide, as well as the rest of our docs, are open-source and available on GitHub. We welcome your contributions.