Migrate from server v3.x to v4.x
On This Page
Migrating from server v3.x to v4.0.x is an in-place migration. You will generate a helm-value file, which will be used in a helm command to upgrade CircleCI server from v3.x to v4.0.x.
We recommend using a staging environment before completing this process in a production environment. This will allow you to gain a better understanding of the migration process, and give you an idea of how long the migration will take to complete.
Your current CircleCI server installation is v3.x.
You have taken a backup of the v3.x instance, following the instructions at the Backup and Restore guide.
If you are using external datastores, they need to be backed up separately.
If you are using internal datastores they are backed up using the backup process described in the link above.
The migration script must be run from a machine with following tools installed:
To perform an in-place upgrade of CircleCI server from v3.x to v4.x, you will execute a migration script, which will perform the following tasks:
Export the KOTS config values into a YAML file.
Modify the exported YAML file to be used with
helmto migrate to server v4.x.
Annotate all the kubernetes resources for
Execute postgres db
Execute cleanup steps which will delete all the KOTS related resources.
Display an output message with the next steps.
| In CircleCI server v4.x, we have changed the way we store workflow data. Previously we had stored your workflow data in Postgres. We have moved this data into your storage bucket (GCS/S3/MinIO). After migrating, there will be a period where your workflow data will be unavailable. This is due to your existing workflow data being transferred to your storage bucket. The time taken depends on how much existing workflow data you have, however, you may scale the |
The kots-exporter script can also perform the following functions if required:
Rerun the annotation step only (
Run the kots-annotation cleanup job (
Display the output message (
Rerun the PostgreSQL database migration (
1. Create docker secret for CircleCI image registry
The migration script requires access to the CircleCI image registry to run the migration function. The image registry is also used in the server v4.x installation. Create the
docker-registry secret in the same Kubernetes namespace in-which CircleCI server is installed. Image registry credentials will be provided by your CircleCI point of contact.
In a terminal run the following, substituting all values indicated by
kubectl -n <namespace> create secret docker-registry regcred \ --docker-server=https://cciserver.azurecr.io \ --docker-username=<image-registry-username> \ --docker-password=<image-registry-token> \ --docker-email=<notification-email-id>
2. Clone the repository and run the kots-exporter script
The instructions below guide you through cloning the repository containing the KOTS exporter script. This script generates the
helm-value file for server 4.x.
git clone https://github.com/CircleCI-Public/server-scripts.
Change into the
Run the migration script:
You will be prompted for the following information:
Release name of your server 3.x installation -
Kubernetes namespace of your server 3.x installation
License String for Server 4.x, which will be provided by your CircleCI point of contact.
After the script has completed, you will get a message detailing the next steps to perform, for example:
############ Upgrade Commands ################ Before upgrading to 4.x, follow the steps below: Your helm values file with your installation config is here: - <path>/kots-exporter/output/helm-values.yaml ## Helm login to cciserver.azurecr.io helm registry login cciserver.azurecr.io --username <image-registry-username> ## Helm Diff (optional) The Helm Diff tool is used to verify that the changes between your current install and the upgrade are expected. # diff command helm diff upgrade <release-name> -n <namespace> -f <path>/kots-exporter/output/helm-values.yaml --show-secrets --context 5 oci://cciserver.azurecr.io/circleci-server --version 4.0.5 ------------------------------------------------------------------------- ## Helm Upgrade CircleCI Server helm upgrade <release-name> -n <namespace> -f <path>/kots-exporter/output/helm-values.yaml oci://cciserver.azurecr.io/circleci-server --version 4.0.5 --force
|After server 3.x to 4.x migration, You must rerun the Nomad terraform with modified value of 'server_endpoint' variable It should be - <domain-name>:4647|
|When upgrading to CircleCI Server v4.0, if your Postgres instance is not externalized, the Postgres chart is also upgraded.|
3. Check migrator job status
Check that the migrator job exists:
kubectl -n <namespace> get job/circle-migrator
If the job does not exist, check the pod’s logs (
circle-migrator.log) in the
log directory. An example of the logs can be found below:
Successfully baselined schema with version: 1 Current version of schema "public": 1 DEBUG: Parsing V0002__drop_domain_migrations.sql ... DEBUG: Found statement at line 1: DROP TABLE IF EXISTS public.domain_migrations DEBUG: Starting migration of schema "public" to version "0002 - drop domain migrations" ... Migrating schema "public" to version "0002 - drop domain migrations" DEBUG: Executing SQL: DROP TABLE IF EXISTS public.domain_migrations DEBUG: 0 rows affected DEBUG: Successfully completed migration of schema "public" to version "0002 - drop domain migrations" DEBUG: Schema History table "public"."schema_version" successfully updated to reflect changes Successfully applied 1 migration to schema "public", now at version v0002 (execution time 00:00.101s) DEBUG: Memory usage: 12 of 15M
4. Validate your helm-value file
Once the migration script has completed, a
helm-values.yaml file will be generated with your existing CircleCI Server 3.x config. This file holds the configuration data you had previously entered in KOTS. Going forward, you will use this file to update/configure your CircleCI Server installation as is standard helm practice.
5. Generate helm-diff output
Next, execute the helm-diff command and review the output.
helm registry login cciserver.azurecr.io -u <image-registry-username> helm diff upgrade <release-name> -n <namespace> -f <path>/kots-exporter/output/helm-values.yaml --show-secrets --context 5 oci://cciserver.azurecr.io/circleci-server --version 4.0.5
Review the output generated from the
helm-diff command using the following to help:
Yellow: Kubernetes resources status, for example,
Red: Deletion, for example,
line displayed in
Green: Addition, for example,
Below are the changes you should expect to see in
imagePullSecretsis added into all the Kubernetes resources
Container images are updated
Secret environment variables (for example api-token, signing-keys) now reference Kubernetes secrets
Environment variables for RabbitMQ and MongoDB URIs change
Environment variables for VM, OUTPUT and NOMAD service uri now reference
Annotations from VM, OUTPUT and NOMAD service resources are deleted
GitHub checksum is added as annotation
Secret and annotation for
distributor-*deployments are deleted
Upsteam charts will be recreated (delete and create):
Prometheus (circleci-server-kube-state-metrics, node-exporter, prometheus-server)
Redis (redis-master, redis-slave)
6. Upgrading CircleCI Server 3.x
Once your helm-value file is verified, run the following commands to upgrade the CircleCI server to v4.x.
Our helm registry is stored in an azure private registry. You will be provided a username and token to access the registry.
helm upgrade circleci-server -n <namespace> -f <path>/kots-exporter/output/helm-values.yaml oci://cciserver.azurecr.io/circleci-server --version 4.0.5 --force
7. Check upgrade status
Run the following command to check all pods are up and running:
kubectl -n <namespace> get pods
8. Update DNS setting
Server 4.x migration is a destructive change for your DNS configuration. Server 4.x replaces the need for 4 load balancers and 5 DNS records with a single`load-balancer/external-ip` service, named
circleci-proxy-acm. This load balancer only needs to be routed via 2 DNS records, <your-domain> and app.<your-domain>. Separate domains for vm-service, output-processer and nomad are no longer required. Retrieve the external IP/Loadbalancer and update your DNS records accordingly.
kubectl -n <namespace> get svc circleci-proxy # AWS Provider: XXXXX.elb.XXXXX.amazonaws.com # GCP Provider: XXX.XXX.XXX.XXX
The following Kubernetes service objects are renamed/changed:
circleci-server-traefik (LoadBalancer) → kong (ClusterIP)
nomad-server-external (LoadBalancer) → nomad-server (ClusterIP)
output-processor (LoadBalancer) → output-processor (ClusterIP)
vm-service (LoadBalancer) → vm-service (ClusterIP)
The following Kubernetes service object is added:
circleci-proxy or circleci-proxy-acm (LoadBalancer)
9. Execute Nomad Terraform
10. Validate your migration to server v4.x
Re-run realitycheck on your new server 4.x environment by pushing a fresh commit.
11. Update your team
Once you have successfully run realitycheck, notify your team about the upgrade.
Help make this document better
This guide, as well as the rest of our docs, are open source and available on GitHub. We welcome your contributions.
- Suggest an edit to this page (please read the contributing guide first).
- To report a problem in the documentation, or to submit feedback and comments, please open an issue on GitHub.
- CircleCI is always seeking ways to improve your experience with our platform. If you would like to share feedback, please join our research community.
Our support engineers are available to help with service issues, billing, or account related questions, and can help troubleshoot build configurations. Contact our support engineers by opening a ticket.
You can also visit our support site to find support articles, community forums, and training resources.
CircleCI Documentation by CircleCI is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.