Backup and restore
Overview
While operating and administering CircleCI server, you will need to consider how to maintain backups and recover your installation, should there be a need to migrate it to another cluster or recover from a critical event.
CircleCI recommends Velero for backup and restore. The benefit of this approach is that it not only restores your application’s data, but it also restores the state of the Kubernetes cluster and its resources at the time of the backup. CirleCI server supports backup and restore with Velero 1.12. This document outlines recommendations for how to back up and restore your CircleCI server instance data and state using Velero.
| Backup and restore of the CircleCI services is dependent on Velero. If your cluster is lost, you will not be able to restore CircleCI until you have successfully started Velero in the cluster. From there you can recover the CircleCI services. |
Velero installation
To use Velero, you must first install the Velero CLI. Then using Velero’s CLI, you will need to install the Velero server components along with the appropriate plugin for your cloud storage solution. Follow the instructions linked below based on your chosen storage backend:
CircleCI server makes use of Persistent Volumes for data storage. To include these volumes in your backups you will need to add the node agent flag --use-node-agent to your Velero install command. Read here for more information. |
Creating backups
Now that Velero is installed on your cluster, you are ready to create your first backup.
K8S_NS=<your_server_namespace>
CHART=$(helm list -n "$K8S_NS" -o yaml | yq '.[].chart' )
REV=$(helm list -n "$K8S_NS" -o yaml | yq '.[].revision')
RANDOM_STR=$(cat /dev/urandom | env LC_ALL=C tr -dc 'a-z0-9' | head -c 8)
velero backup create "${K8S_NS}-${RANDOM_STR}" --include-namespaces "${K8S_NS}" --labels "chart--rev=${CHART}--${REV}"Restoring backups
Below are the instructions for how to restore your CircleCI server install using Velero.
# List all existing backups
velero backup get --show-labels
# Restore the specific backup
velero restore create --include-namespaces <circleci-namespace> --from-backup <backup-name>Scheduling backups
See Velero’s documentation on creating scheduled backups.
Troubleshooting
Errors occur during backup or restore process
If you experience an error during backup or restore processes, the first place to look would be the Velero logs. Using the command below, you may find 4XX errors, which would likely be caused by issues with your storage bucket access.
-
Confirm that your bucket exists and is in the region you expect.
-
Confirm that the credentials provided to Velero can be used to access the bucket.
-
You may need to run the command to install Velero again, this time with updated bucket information.
You may also check the status of pods in the velero namespace:
$ kubectl get pods --namespace velero
NAME READY STATUS RESTARTS AGE
node-agent-5vlww 1/1 Pending 0 10m
node-agent-94ptv 1/1 Running 0 10m
node-agent-ch6m9 1/1 Pending 0 10m
node-agent-mknws 1/1 Running 0 10m
velero-68788b675c-dm2s7 1/1 Running 0 10mIn the above example, some Restic pods are pending, which means they are waiting for a node to have available CPU or memory resources. In this case, you may need to scale your nodes to accommodate Restic.
For more details on troubleshooting Velero issues, refer to the Velero documentation.