Start Building for Free
CircleCI.comAcademyBlogCommunitySupport

Backup and restore

2 months ago1 min read
Server v4.x
Server Admin
On This Page

Overview

While operating and administering CircleCI server, you will need to consider how to maintain backups and recover your installation, should there be a need to migrate it to another cluster or recover from a critical event.

This document outlines recommendations for how to back up and restore your CircleCI server instance data and state. For setup instructions see Phase 4 – Post installation.

CircleCI recommends Velero for backup and restore. The benefit of this approach is that it not only restores your application’s data, but it also restores the state of the Kubernetes cluster and its resources at the time of the backup. Setup and install of Velero is covered in the server v4.x installation guide.

Creating backups

K8S_NS=$(helm list -o yaml  | yq '.[].namespace')
CHART=$(helm list -o yaml  | yq '.[].chart' )
REV=$(helm list -o yaml  | yq '.[].revision')
RANDOM_STR=$(cat /dev/urandom | env LC_ALL=C tr -dc 'a-z0-9' | head -c 8)

velero backup create "${K8S_NS}-${RANDOM_STR}" --include-namespaces "${K8S_NS}" --labels "chart--rev=${CHART}--${REV}"

Restoring backups

# List all existing backups
velero backup get --show-labels

# Restore the specific backup
velero restore create --include-namespaces <circleci-namespace> --from-backup <backup-name>

Scheduling backups

See Velero’s documentation on creating scheduled backups.

Troubleshooting

Errors occur during backup or restore process

If you experience an error during backup or restore processes, the first place to look would be the Velero logs. Using the command below, you may find 4XX errors, which would likely be caused by issues with your storage bucket access.

  • Confirm that your bucket exists and is in the region you expect.

  • Confirm that the credentials provided to Velero can be used to access the bucket.

  • You may need to run the command to install Velero again, this time with updated bucket information.

You may also check the status of pods in the velero namespace:

$ kubectl get pods --namespace velero
NAME                      READY   STATUS    RESTARTS   AGE
restic-5vlww              1/1     Pending   0          10m
restic-94ptv              1/1     Running   0          10m
restic-ch6m9              1/1     Pending   0          10m
restic-mknws              1/1     Running   0          10m
velero-68788b675c-dm2s7   1/1     Running   0          10m

In the above example, some restic pods are pending, which means they are waiting for a node to have available CPU or memory resources. In this case, you may need to scale your nodes to accommodate restic.


Help make this document better

This guide, as well as the rest of our docs, are open source and available on GitHub. We welcome your contributions.

Need support?

Our support engineers are available to help with service issues, billing, or account related questions, and can help troubleshoot build configurations. Contact our support engineers by opening a ticket.

You can also visit our support site to find support articles, community forums, and training resources.