CircleCI Server v2.19 to v4.0 Migration

Server v4.x

Server Admin

Migrating from v2.19.x to v4.0 requires you to back up your v2.19 instance data (including Mongo, Postgres, and Vault), and then restore that data in a waiting server v4.0 instance. If you experience problems, you can fall back to your v2.19 instance. Migration requires an operational server v4.0 installation.

Depending on the size of your data stores, the migration can take anywhere from a few minutes to a few hours. Using a staging environment before completing this process in a production environment is recommended. Using a staging environment not only allows you to gain a better understanding of the migration process, but also gives you an idea of how long the migration will take to complete.

Prerequisites

Your current CircleCI server installation is v2.19.x
You have taken a backup of your v2.19 instance. If you are using external datastores, they need to be backed up separately.
If your installation is in a Google Cloud Platform (GCP) environment, you must use Workload Identities, not Service Account credentials. This is due to a limitation in init container volume mounting. No such limitation exists in AWS environments.
Datastores:
1. If using externalized datastores, Postgres must be updated to version 12.
2. If your datastores are internal, ensure that the volume sizes allocated to Mongodb, Postgres, and Redis in your v4.0 installation are at least the same or greater than the volumes allocated to the Mongodb, Postgres, and Redis instances in your v2.19 install. You may resize your 4.x datastore volumes following these instructions.
You have a new CircleCI server v4.0.x installation.
You have successfully run realitycheck with contexts before starting.
The migration script must be run from a machine with:
- kubectl configured for the server v4.0 instance
- ssh access to the v2.19 services machine

Migration

Migrating to server v4.0 will shut down your v2.19 application. Your v2.19 application will not be restarted, although you may manually restart it using the replicated Admin Console. Running server v2.19 and server v4.0 at the same time can cause issues with your 2.19 build data. Server v2.19 should NOT be restarted if server v4.0 is running.

Starting the migration process will cause downtime. Scheduling a maintenance window is recommended.

The instructions below will clone the repository containing the server v2.19.x to server v4.0 migration script.

The migration script will:

Stop your v2.19.x application.
Perform preflight checks to confirm namespace and datastores for v2.19.x.
Create a tarball of your v2.19.x application’s PostgreSQL and Mongo databases.
Archive existing application data for Vault and CircleCI encryption/signing keys.
Export the v2.19.x tarball to your v4.0 installation. Exported data stores are stored in a directory named circleci_export, located relative to wherever the migration script is run from. This can be useful for debugging purposes.
Perform preflight checks to confirm namespace and datastores for your v4.0 instance.
Scale v4.0 application deployments down to zero.
Import the data from the previously exported tarball to your new v4.0 instance.
Scale v4.0 application deployments up to one.

1. Clone the repository and run the migration script

In a terminal:

Run git clone https://github.com/CircleCI-Public/server-scripts.
Change into the migrate directory: cd server-scripts/migrate.
Run the migration script: ./migrate.sh --server4. NOTE: If you have externalized services, you can run bash migrate.sh --server4 -v -p -m. These -v -p -m flags will skip the migration of Vault, Postgres, and Mongo, respectively. Skipping all three will copy your keys from /data/circle/circleci-encryption-keys on the v2.19.x services machine, allowing you to cat these files and upload their contents to the v4.0 configuration page.
You will be prompted for the following information:
- Username of your server 2.19 installation
- Hostname of your server 2.19 installation
- The path to your SSH key file for your server 2.19.x installation
- Kubernetes namespace of your server 4.0 installation
After the script has completed, the Signing and Encryption keys from the 2.19 instance will need to be added to the new 4.0 instance via helm values file or updating the secret used for your 4.0 encryption keys. The keys will be located in circleci_export/circle-data.
The 4.0 instance will either need to be updated to point at the same storage bucket that the 2.19 instance used, or the data needs to be copied over to a new bucket. The latter option ensures the 2.19 instance continues to work as expected, and so is the recommended approach if this migration is part of a test.

If a different hostname is being used in the 4.0 environment, the GitHub webhooks will still be pointing to the hostname used in the 2.19 environment. The easiest way to update this is to click Stop Building and then Set Up Project. After doing this, the contexts and environment variables associated with the project will still be present.

2. Validate your migration to server v4.0

Re-run realitycheck with contexts on your new server 4.0 environment by pushing a fresh commit.

3. Update your team

Once you have successfully run realitycheck, notify your team of the new CircleCI UI and URL, if it has changed.

Frequently Asked Questions

Where did all my job and build history go?

All of your existing jobs and build history have been moved to the Legacy Jobs view. You can view the complete job history using one of the following methods:
- Selecting Projects → PROJECT_NAME and selecting the legacy jobs view link at the bottom of the project’s build history
- Using the following URL pattern: https://<APP_DOMAIN>/pipelines/github/<ORG>/<PROJECT>/jobs
- For a specific job, append a job number to the URL: https://<APP_DOMAIN>/pipelines/github/<ORG>/<PROJECT>/jobs/<JOB_NUMBER>

Why does nothing happen when I select "Start Building" on my project after migration?

By default, a newly added project (a project that has never been followed) triggers a build automatically after it has been followed for the first time. If the project was or ever has been followed in 2.19 or 4.0, it will not be considered a new project or first build and a build will not be triggered after a follow. To trigger a build, perform an activity that triggers a GitHub webhook such as pushing up a new commit or branch.

I got an error: "Error from server (NotFound):"

The script assumes specific naming patterns for your Postgres and MongoDB. If you get this error, it may indicate a non-standard installation, a missing DB migration, or other issues. In this case, contact support with a support bundle and the output from the migration script.

Transitioning to pipelines

When migrating from a server v2.x to a v4.0 installation you will have project configurations made before the introduction of pipelines. Pipelines are automatically enabled for server v4.0 installations so all you need to do is change your project configurations (.circleci/_config.yml) to version: 2.1 to access all CircleCI features available for server v4.