CircleCI Server v3.x Internal Database Volume Expansion

Overview

Persistent volume expansion of MongoDB and Postgres is available for server v3.2.0 and up.

If you have chosen to deploy either of the CircleCI databases (MongoDB or Postgres) within the cluster, as opposed to externally provisioning these databases, then there may come a point at which the storage space initially made available to these databases is no longer sufficient. Internal databases in your Kubernetes cluster make use of persistent volumes for persistent storage. The size of these volumes is determined by persistence volume claims (PVCs). These PVCs request storage space based on what has been made available to the nodes in your cluster.

This document runs through the steps required to increase PVCs in order to expand the space available to your internally deployed databases. This operation should not require any downtime unless you need to restart your database pods.

Expanding persistent volumes does not affect the size of the storage attached to your nodes. Expanding node storage remains within the limitations of your cloud provider. Please refer to the docs for your chosen cloud provider for details on how to expand the storage attached to your cluster’s nodes.

Resizing persistent volume claims

Below are the steps detailing how to resize the persistent volume claims for Postgres and MongoDB. We will confirm the size of the claims and the disk space made available to our databases before and after this operation.

As a precaution, it’s always good to create a backup of your cluster first.

Step 0 - Confirm current volume size

By default, the persistent volume claims used by our internal databases have a capacity of 8Gi. However, this initial value can be set at the time of first deployment from the Kots admin console. We can confirm the size of our persistent volume claim capacity using kubectl get pvc <pvc-name>.

For postgres:

circleci-user ~ $ kubectl get pvc data-postgresql-0
NAME                STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-postgresql-0   Bound    pvc-c2a2d97b-2b7d-47d3-ac77-d07c76c995a3   8Gi        RWO            gp2            1d

For mongodb:

circleci-user ~ $ kubectl get pvc datadir-mongodb-0
NAME                STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
datadir-mongodb-0   Bound    pvc-58a2274c-31c0-487a-b329-0062426b5535   8Gi        RWO            gp2            1d

We can also confirm this capacity is made available to a database by checking the size of its data directory.

For postgres the directory is /bitnami/postgresql. We can confirm its size using the command below.

circleci-user ~ $ kubectl exec postgresql-0 -- df -h /bitnami/postgresql
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme4n1    7.8G  404M  7.4G   3% /bitnami/postgresql

For mongodb the directory is /bitnami/mongodb.

circleci-user ~ $ kubectl exec mongodb-0 -- df -h /bitnami/mongodb
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme1n1    7.8G  441M  7.4G   3% /bitnami/mongodb

From the examples above, the capacities are still 8Gi. The following steps show how to increase this to 10Gi.

Step 1 - Confirm volume expansion is allowed

First, confirm that volume expansion is allowed in your cluster.

circleci-user ~ $ kubectl get sc
NAME            PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
gp2 (default)   kubernetes.io/aws-ebs   Delete          WaitForFirstConsumer   false                  1d

As we can see, our default storage class does not allow volume expansion. However, we can change this with the kubectl patch command.

circleci-user ~ $ kubectl patch sc gp2 -p '{"allowVolumeExpansion": true}'
storageclass.storage.k8s.io/gp2 patched
circleci-user ~ $ kubectl get sc
NAME            PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
gp2 (default)   kubernetes.io/aws-ebs   Delete          WaitForFirstConsumer   true                  1d

Now we may proceed to expand our volumes.

Step 2 - Delete the database’s stateful set

In this step we will delete the stateful set, which controls our database pod. The command below will delete the referenced database’s stateful set without deleting the pod. We do not want to delete the pod itself as this would cause downtime. In the following steps we will be redeploying our stateful set. You might chose to delete one or both stateful sets, depending on which database volumes you wish to expand. The --cascade=false flag is most important here.

For postgres:

circleci-user ~ $ kubectl delete sts postgresql --cascade=false

For mongodb:

circleci-user ~ $ kubectl delete sts mongodb --cascade=false

Step 3 - Update the size of the database’s PVC

Now that the stateful set has been removed, we can increase the size of our persistent volume claim to 10Gi.

For postgres:

circleci-user ~ $ kubectl patch pvc data-postgresql-0 -p '{"spec": {"resources": {"requests": {"storage": "10Gi"}}}}'

For mongodb:

circleci-user ~ $ kubectl patch pvc datadir-mongodb-0 -p '{"spec": {"resources": {"requests": {"storage": "10Gi"}}}}'

Step 4 - Update KOTS admin with the new PVC size

Now we need to head over to the KOTS admin console to persist our changes. In the config section, we will update the values for our PVC size to 10Gi as shown below.

Postgres PVC size
Figure 1. Postgres
MongoDB PVC size
Figure 2. MongoDB

Now save and deploy your changes. This will recreate the stateful set(s) that we destroyed earlier, but with the new PVC sizes which will persist through new releases.

Step 5 - Validate new volume size

Once deployed, we can validate the size of the data directories assigned to our databases.

For postgres the directory is /bitnami/postgresql.

circleci-user ~ $ kubectl exec postgresql-0 -- df -h /bitnami/postgresql
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme4n1    9.8G  404M  9.4G   5% /bitnami/postgresql

For mongodb the directory is /bitnami/mongodb.

circleci-user ~ $ kubectl exec mongodb-0 -- df -h /bitnami/mongodb
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme1n1    9.8G  441M  9.3G   5% /bitnami/mongodb

As we can see, the size of our directories has been increased.

If you find that after following these steps, the disk size allocated to your data directories have not increased, then you may need to restart your database pods. This, however, will cause some downtime of 1-5mins as the databases restart. You can use the commands below to restart your databases.

For postgres

circleci-user ~ $ kubectl rollout restart sts postgresql

For mongodb

circleci-user ~ $ kubectl rollout restart sts mongodb


Help make this document better

This guide, as well as the rest of our docs, are open-source and available on GitHub. We welcome your contributions.