CircleCI Server v3.x Migration

Migrating from 2.19.x to 3.x requires you to back up your 2.19 instance data (Mongo, Postgres, and Vault) and then restore that data in a waiting Server 3.x instance. If you run into trouble you can fallback to your 2.19 instance. Migration does require an already operating Server 3.x installation. Depending on the size of your data stores, the migration can take anywhere from a few minutes to a few hours. We recommend using a staging environment before completing this process in a production environment. This will not only allow you to gain a better understanding of the migration process, but will also give you a feel for how long the migration will take to complete.

Prerequisites

  1. Your current CircleCI Server installation is 2.19.

  2. You have taken a backup of the 2.19 instance. If you are using external datastores, they will need to be backed up separately.

  3. You have a new CircleCI Server 3.x installation.

  4. You have successfully run reality check with contexts prior to starting.

  5. The migration script must be run from a machine with:

    • kubectl configured for the server 3.x instance

    • ssh access to the 2.19 services box

External Datastores Only

  1. Backups have been taken of all external data stores.

  2. Postgres has been updated to version 12.

Internal Datastore Only

  1. You have taken a backup of the 2.19 instance.

  2. You have successfully run reality check on the new server 3.x instance with contexts prior to starting.

Migration

Migrating to server v3.x will shut down your v2.19 application. Your v2.19 application will not be started back up, although you may manually start it back up using the administrative console.
Starting the migration process will cause downtime. It is recommended you schedule a maintenance window.
Running server 2.19 and server 3.x at the same time can cause issues to your 2.19 build data. Server 2.19 should NOT be restarted if server 3.x is running.

Step 1 - Clone the repository and run the migration script

The instructions below will clone the repository containing the server v2.19.x to server v3.x migration script. The migration script will:

  • Stop your v2.19.x application

  • Perform pre-flight checks to confirm namespace and datastores for 2.19.x.

  • Create a tarball of your v2.19.x application’s PostgreSQL and Mongo databases.

  • Archive existing application data for Vault and CircleCI encryption/signing keys.

  • Export the 2.19.x tarball to your v3.x installation. Exported data stores are stored in a directory named circleci_export, located relative to wherever the migration script is run from. This can be useful for debugging purposes.

  • Perform pre-flight checks to confirm namespace and datastores for 3.x instance.

  • Scale v3.x application deployments down to zero.

  • Import the data from the previously exported tarball to your new v3.x instance.

  • Scale v3.x application deployments up to one.

If you have externalized services then you can run bash migrate.sh -v -p -m. These -v -p -m flags will skip the migration of Vault, Postgres, and Mongo, respectively. Skipping all three will copy your keys from /data/circle/circleci-encryption-keys on the v2.19.x services machine, allowing you to cat these files and upload their contents to the 3.x configuration page.

In a terminal:

  1. Run git clone https://github.com/CircleCI-Public/server-scripts.

  2. Change into the migrate directory: cd server-scripts/migrate.

  3. Run the migration script: ./migrate.sh.

  4. You will be prompted for the following information:

    • Username of your server 2.19 installation

    • Hostname of your server 2.19 installation

    • The path to your SSH key file for your server 2.19.x installation

    • Kubernetes namespace of your server 3.x installation

  5. After the script has completed, the Signing and Encryption keys from the 2.19 instance will need to be added to the new 3.0 instance via the KOTS Admin Console. The keys will be located in circleci_export/circle-data.

  6. The 3.x instance will either need to be updated to point at the same storage bucket that the 2.19 instance used, or the data needs to be copied over to a new bucket. The latter will ensure the 2.19 instance continues to work as expected, and so is the recommended approach if this migration is part of a test.

If a different hostname is being used in the 3.x environment, the GitHub webhooks will still be pointing to the hostname used in the 2.19 environment. The easiest way to update this is to click Stop Building and then Set Up Project. After doing this, the contexts and environment variables associated with the project will still be present.

Step 2 - Validate your migration to Server 3.0

Re-run reality check with contexts on your new server 3.x environment by pushing a fresh commit.

Step 3 - Update your team

Once you have successfully run reality check, notify your team of the new CircleCI UI and URL, if it has changed.

Frequently Asked Questions

Where did all my job and build history go?

Why does nothing happen when I select "Start Building" on my project after migration?

  • By default, a newly added project (a project that has never been followed) will trigger a build automatically after it has been followed for the first time. If the project was or ever has been followed in 2.0 or 3.0, it will not be considered a new project or first build and a build will not be triggered after follow. To trigger a build, perform an activity that will trigger a Github webhook such as pushing up a new commit or branch.

I got an error "Error from server (NotFound):"

  • The script assumes specific naming patterns for your Postgres and MongoDB. If you get this error, it may indicate a non-standard installation, a missing DB migration, or other issues. In this case it is best to contact support with a support bundle and the output from the migration script.



Help make this document better

This guide, as well as the rest of our docs, are open-source and available on GitHub. We welcome your contributions.