Start Building for Free
CircleCI.comAcademyBlogCommunitySupport

Migrate from server v3.x to v4.x

5 days ago5 min read
Server v4.x
Server Admin
On This Page

Migrating from server v3.x to v4.x is an in-place migration. You will generate a helm-value file, which will be used in a helm command to upgrade the CircleCI server from v3.x to v4.x.

We recommend using a staging environment before completing this process in a production environment. This will allow you to gain a better understanding of the migration process, and give you an idea of how long the migration will take to complete.

Prerequisites

Migration

To perform an in-place upgrade of CircleCI server from v3.x to v4.x, you will execute a migration script, which will perform the following tasks:

  • Export the KOTS config values into a YAML file.

  • Modify the exported YAML file to be used with helm to migrate to server v4.x.

  • Annotate all the kubernetes resources for helm.

  • Execute postgres db domain migration.

  • Execute cleanup steps which will delete all the KOTS related resources.

  • Display an output message with the next steps.

The kots-exporter script can also perform the following functions if required:

  • Rerun the annotation step only (-f annotate)

  • Run the kots-annotation cleanup job (-f kots_cleanup)

  • Display the output message (-f message)

  • Rerun the PostgreSQL database migration (-f flyway)

1. Create docker secret for CircleCI image registry

The migration script requires access to the CircleCI image registry to run the migration function. The image registry is also used in the server v4.x installation. Create the docker-registry secret in the same Kubernetes namespace in-which CircleCI server is installed. Image registry credentials will be provided by your CircleCI point of contact.

In a terminal run the following, substituting all values indicated by < >:

kubectl -n <namespace> create secret docker-registry regcred \
  --docker-server=https://cciserver.azurecr.io \
  --docker-username=<image-registry-username> \
  --docker-password=<image-registry-token> \
  --docker-email=<notification-email-id>

2. Clone the repository and run the kots-exporter script

The instructions below guide you through cloning the repository containing the KOTS exporter script. This script generates the helm-value file for server 4.x.

  1. Run git clone https://github.com/CircleCI-Public/server-scripts.

  2. Change into the kots-exporter directory: cd server-scripts/kots-exporter.

  3. Run the migration script: ./kots-exporter.sh.

  4. You will be prompted for the following information:

    • Release name of your server 3.x installation - circleci-server

    • Kubernetes namespace of your server 3.x installation

    • License String for Server 4.x, which will be provided by your CircleCI point of contact.

  5. After the script has completed, you will get a message detailing the next steps to perform, for example:

############ Upgrade Commands ################
Before upgrading to 4.x, follow the steps below:

Your helm values file with your installation config is here:
- <path>/kots-exporter/output/helm-values.yaml

## Helm login to cciserver.azurecr.io
helm registry login cciserver.azurecr.io --username <image-registry-username>

## Helm Diff (optional)
The Helm Diff tool is used to verify that the changes between your current install and the upgrade are expected.

# diff command
helm diff upgrade <release-name> -n <namespace> -f <path>/kots-exporter/output/helm-values.yaml --show-secrets --context 5 oci://cciserver.azurecr.io/circleci-server --version 4.0.1
-------------------------------------------------------------------------
## Helm Upgrade CircleCI Server
helm upgrade <release-name> -n <namespace> -f <path>/kots-exporter/output/helm-values.yaml oci://cciserver.azurecr.io/circleci-server --version <version-to-upgrade-to> --force

NOTE: After server 3.x to 4.x migration, You must rerun the Nomad terraform with modified value of 'server_endpoint' variable
It should be - <domain-name>:4647

3. Check migrator job status

Check that the migrator job exists:

kubectl -n <namespace> get job/circle-migrator

If the job does not exist, check the pod’s logs (circle-migrator.log) in the log directory. An example of the logs can be found below:

Successfully baselined schema with version: 1
Current version of schema "public": 1
DEBUG: Parsing V0002__drop_domain_migrations.sql ...
DEBUG: Found statement at line 1: DROP TABLE IF EXISTS public.domain_migrations
DEBUG: Starting migration of schema "public" to version "0002 - drop domain migrations" ...
Migrating schema "public" to version "0002 - drop domain migrations"
DEBUG: Executing SQL: DROP TABLE IF EXISTS public.domain_migrations
DEBUG: 0 rows affected
DEBUG: Successfully completed migration of schema "public" to version "0002 - drop domain migrations"
DEBUG: Schema History table "public"."schema_version" successfully updated to reflect changes
Successfully applied 1 migration to schema "public", now at version v0002 (execution time 00:00.101s)
DEBUG: Memory usage: 12 of 15M

4. Validate your helm-value file

Once the migration script has completed, a helm-values.yaml file will be generated with your existing CircleCI Server 3.x config. This file holds the configuration data you had previously entered in KOTS. Going forward, you will use this file to update/configure your CircleCI Server installation as is standard helm practice.

5. Generate helm-diff output

Next, execute the helm-diff command and review the output.

helm registry login cciserver.azurecr.io -u <image-registry-username>

helm diff upgrade <release-name> -n <namespace> -f <path>/kots-exporter/output/helm-values.yaml --show-secrets --context 5 oci://cciserver.azurecr.io/circleci-server --version 4.0.1

Review the output generated from the helm-diff command using the following to help:

  • line highlighted Yellow: Kubernetes resources status, for example, changed, added

  • line highlighted Red: Deletion, for example, image

  • line displayed in Green: Addition, for example, imagePullSecret

Below are the changes you should expect to see in helm-diff output:

  • imagePullSecrets is added into all the Kubernetes resources

  • Container images are updated

  • Secret environment variables (for example api-token, signing-keys) now reference Kubernetes secrets

  • Environment variables for RabbitMQ and MongoDB URIs change

  • Environment variables for VM, OUTPUT and NOMAD service uri now reference <domain_name>:<service_port>

  • Annotations from VM, OUTPUT and NOMAD service resources are deleted

  • Github checksum is added as annotation

  • Secret and annotation for distributor-* deployments are deleted

  • Upstream chart postgresql is updated

  • Upsteam charts will be recreated (delete and create):

    • Prometheus (circleci-server-kube-state-metrics, node-exporter, prometheus-server)

    • MongoDB

    • RabbitMQ

    • Redis (redis-master, redis-slave)

6. Upgrading CircleCI Server 3.x

Once your helm-value file is verified, run the following commands to upgrade the CircleCI server to v4.x.

Our helm registry is stored in an azure private registry. You will be provided a username and token to access the registry.

helm upgrade circleci-server -n <namespace> -f <path>/kots-exporter/output/helm-values.yaml oci://cciserver.azurecr.io/circleci-server --version 4.0.1 --force

7. Check upgrade status

Run the following command to check all pods are up and running:

kubectl -n <namespace> get pods

8. Update DNS setting

Server 4.x migration is a destructive change for your DNS configuration. Server 4.x replaces the need for 4 load balancers and 5 DNS records with a single`load-balancer/external-ip` service, named circleci-proxy or circleci-proxy-acm. This load balancer only needs to be routed via 2 DNS records, <your-domain> and app.<your-domain>. Separate domains for vm-service, output-processer and nomad are no longer required. Retrieve the external IP/Loadbalancer and update your DNS records accordingly.

kubectl -n <namespace> get svc circleci-proxy

# AWS Provider: XXXXX.elb.XXXXX.amazonaws.com
# GCP Provider: XXX.XXX.XXX.XXX

The following Kubernetes service objects are renamed/changed:

  • circleci-server-traefik (LoadBalancer) → kong (ClusterIP)

  • nomad-server-external (LoadBalancer) → nomad-server (ClusterIP)

  • output-processor (LoadBalancer) → output-processor (ClusterIP)

  • vm-service (LoadBalancer) → vm-service (ClusterIP)

The following Kubernetes service object is added:

  • circleci-proxy or circleci-proxy-acm (LoadBalancer)

9. Execute Nomad Terraform

Execute the Nomad Terraform to re-create nomad client where server_endpoint value is set to be <domain>:4647. You can follow the steps mentioned here. Update the helm value file with generated Certificates and Keys (base64 encoded) for Nomad Sever-Client communication.

10. Validate your migration to server v4.x

Re-run realitycheck on your new server 4.x environment by pushing a fresh commit.

11. Update your team

Once you have successfully run realitycheck, notify your team about the upgrade.


Help make this document better

This guide, as well as the rest of our docs, are open source and available on GitHub. We welcome your contributions.

Need support?

Our support engineers are available to help with service issues, billing, or account related questions, and can help troubleshoot build configurations. Contact our support engineers by opening a ticket.

You can also visit our support site to find support articles, community forums, and training resources.