Docs
circleci.com
Start Building for Free

Expanding Internal Database Volumes

2 weeks ago3 min read
Server v4.x
Server Admin
On This Page

Overview

If you have chosen to deploy either of the CircleCI databases (MongoDB or PostgreSQL) within the cluster, rather than externally provisioning these databases, there may come a point at which the storage space initially made available to these databases is no longer sufficient. Internal databases in your Kubernetes cluster make use of persistent volumes for persistent storage. The size of these volumes is determined by persistence volume claims (PVCs). These PVCs request storage space based on what has been made available to the nodes in your cluster.

This document runs through the steps required to increase PVCs to expand the space available to your internally deployed databases. This operation should not require any downtime, unless you need to restart your database pods.

Resizing persistent volume claims

Below are the steps detailing how to resize the persistent volume claims for PostgreSQL and MongoDB. You will confirm the size of the claims and the disk space made available to your databases before and after this operation.

Step 0 - Confirm current volume size

By default, the persistent volume claims used by our internal databases have a capacity of 8Gi. However, this initial value can be set at the time of first deployment via the helm-value file. You can confirm the size of your persistent volume claim capacity using the command: kubectl get pvc <pvc-name>.

For PostgreSQL:

circleci-user ~ $ kubectl get pvc data-postgresql-0
NAME                STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-postgresql-0   Bound    pvc-c2a2d97b-2b7d-47d3-ac77-d07c76c995a3   8Gi        RWO            gp2            1d

For MongoDB:

circleci-user ~ $ kubectl get pvc datadir-mongodb-0
NAME                STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
datadir-mongodb-0   Bound    pvc-58a2274c-31c0-487a-b329-0062426b5535   8Gi        RWO            gp2            1d

You can also confirm this capacity is made available to a database by checking the size of its data directory.

For PostgreSQL, the directory is /bitnami/postgresql. You can confirm its size using the command below.

circleci-user ~ $ kubectl exec postgresql-0 -- df -h /bitnami/postgresql
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme4n1    7.8G  404M  7.4G   3% /bitnami/postgresql

For MongoDB, the directory is /bitnami/mongodb.

circleci-user ~ $ kubectl exec mongodb-0 -- df -h /bitnami/mongodb
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme1n1    7.8G  441M  7.4G   3% /bitnami/mongodb

From the examples above, the capacities are still 8Gi. The following steps show how to increase this to 10Gi.

Step 1 - Confirm volume expansion is allowed

First, confirm that volume expansion is allowed in your cluster.

circleci-user ~ $ kubectl get sc
NAME            PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
gp2 (default)   kubernetes.io/aws-ebs   Delete          WaitForFirstConsumer   false                  1d

As you can see, your default storage class does not allow volume expansion. However, you can change this with the kubectl patch command:

circleci-user ~ $ kubectl patch sc gp2 -p '{"allowVolumeExpansion": true}'
storageclass.storage.k8s.io/gp2 patched
circleci-user ~ $ kubectl get sc
NAME            PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
gp2 (default)   kubernetes.io/aws-ebs   Delete          WaitForFirstConsumer   true                  1d

Now you may proceed to expanding your volumes.

Step 2 - Delete the database’s stateful set

In this step, you will delete the stateful set, which controls your database pod. The command below deletes the referenced database’s stateful set without deleting the pod. You do not want to delete the pod itself, as this would cause downtime. In the following steps, you will redeploy your stateful set. You might chose to delete one or both stateful sets, depending on which database volumes you wish to expand. The --cascade=orphan flag is most important here.

For PostgreSQL:

kubectl delete sts postgresql --cascade=orphan

For MongoDB:

kubectl delete sts mongodb --cascade=orphan

Step 3 - Update the size of the database’s PVC

Now that the stateful set has been removed, you can increase the size of our persistent volume claim to 10Gi.

For PostgreSQL:

kubectl patch pvc data-postgresql-0 -p '{"spec": {"resources": {"requests": {"storage": "10Gi"}}}}'

For MongoDB:

kubectl patch pvc datadir-mongodb-0 -p '{"spec": {"resources": {"requests": {"storage": "10Gi"}}}}'

Step 4 - Update helm-value file with the new PVC size

Now you need to upgrade the server installation by modifying the PVC size in the helm-value file to persist your changes. In the helm-value file, you will update the values for your PVC size to 10Gi as shown below.

  • PostgreSQL

    postgresql:
      persistence:
        size: 10Gi
  • MongoDB

    mongodb:
      persistence:
        size: 10Gi

Now save and deploy your changes. This recreates the stateful set(s) that you destroyed earlier, but with the new PVC sizes, which will persist through new releases.

helm upgrade <release-name> -n <namespace> -f < helm-value-file> <chart-dictectory>

Step 5 - Validate new volume size

Once deployed, you can validate the size of the data directories assigned to our databases.

For PostgreSQL the directory is /bitnami/postgresql.

circleci-user ~ $ kubectl exec postgresql-0 -- df -h /bitnami/postgresql
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme4n1    9.8G  404M  9.4G   5% /bitnami/postgresql

For MongoDB the directory is /bitnami/mongodb.

circleci-user ~ $ kubectl exec mongodb-0 -- df -h /bitnami/mongodb
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme1n1    9.8G  441M  9.3G   5% /bitnami/mongodb

As you can see, the size of your directories has been increased.

When completing these steps, if you find, as expected, that the new pods do show the resized volumes, it is still worth checking with the kubectl describe commands shown below. In some instances the resize will fail, but the only way to know is by viewing an event in the output from kubectl describe.

For PostgreSQL:

kubectl describe pvc data-postgresql-0

For MongoDB:

kubectl describe pvc datadir-mongodb-0

Success looks like this:

Events:
Type    Reason                      Age   From     Message

Normal  FileSystemResizeSuccessful  19m   kubelet  MountVolume.NodeExpandVolume succeeded for volume "pvc-b3382dd7-3ecc-45b0-aeff-45edc31f48aa"

Failure might look like this:

Warning  VolumeResizeFailed  58m   volume_expand  error expanding volume "circleci-server/datadir-mongodb-0" of plugin "kubernetes.io/aws-ebs": AWS modifyVolume failed for vol-08d0861715c313887 with VolumeModificationRateExceeded: You've reached the maximum modification rate per volume limit. Wait at least 6 hours between modifications per EBS volume.
status code: 400, request id: 3bd43d1e-0420-4807-9c33-df26a4ca3f23
Normal   FileSystemResizeSuccessful  55m (x2 over 81m)  kubelet        MountVolume.NodeExpandVolume succeeded for volume "pvc-29456ce2-c7ff-492b-add4-fcf11872589f"

Troubleshooting

After following these steps, if you find that the disk size allocated to your data directories has not increased, then you may need to restart your database pods. This will cause downtime of 1-5 minutes while the databases restart. You can use the commands below to restart your databases.

For PostgreSQL:

kubectl rollout restart sts postgresql

For MongoDB:

kubectl rollout restart sts mongodb

Help make this document better

This guide, as well as the rest of our docs, are open source and available on GitHub. We welcome your contributions.

Need support?

Our support engineers are available to help with service issues, billing, or account related questions, and can help troubleshoot build configurations. Contact our support engineers by opening a ticket.

You can also visit our support site to find support articles, community forums, and training resources.