Docs
Start Building for Free
CircleCI.comAcademyBlogCommunitySupport

Expanding Internal Database Volumes

9 months ago3 min read
Server v4.x
Server Admin
On This Page

Overview

If you have chosen to deploy either of the CircleCI databases (MongoDB or PostgreSQL) within the cluster, rather than externally provisioning these databases, there may come a point at which the storage space initially made available to these databases is no longer sufficient. Internal databases in your Kubernetes cluster make use of persistent volumes for persistent storage. The size of these volumes is determined by persistence volume claims (PVCs). These PVCs request storage space based on what has been made available to the nodes in your cluster.

This document runs through the steps required to increase PVCs to expand the space available to your internally deployed databases. This operation should not require any downtime, unless you need to restart your database pods.

Resizing persistent volume claims

Below are the steps detailing how to resize the persistent volume claims for PostgreSQL and MongoDB. You will confirm the size of the claims and the disk space made available to your databases before and after this operation.

1. Confirm current volume size

By default, the persistent volume claims used by our internal databases have a capacity of 8Gi. However, this initial value can be set at the time of first deployment via the helm-value file. You can confirm the size of your persistent volume claim capacity using the command: kubectl get pvc <pvc-name>.

  • PostgreSQL

    circleci-user ~ $ kubectl get pvc data-postgresql-0

NAME                STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-postgresql-0   Bound    pvc-c2a2d97b-2b7d-47d3-ac77-d07c76c995a3   8Gi        RWO            gp2            1d

  • MongoDB

    circleci-user ~ $ kubectl get pvc datadir-mongodb-0

NAME                STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
datadir-mongodb-0   Bound    pvc-58a2274c-31c0-487a-b329-0062426b5535   8Gi        RWO            gp2            1d

  • Redis

    circleci-user ~ $ kubectl get pvc redis-data-redis-master-0

NAME                        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
redis-data-redis-master-0   Bound    pvc-522b3e1c-172d-482c-8648-c24896d18a72   8Gi       RWO            gp2            64m

circleci-user ~ $ kubectl get pvc redis-data-redis-slave-0

NAME                        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
redis-data-redis-slave-0   Bound    pvc-bfdf976d-6f8b-4136-aaa0-b5fc3fce826b   8Gi       RWO            gp2            64m

You can also confirm this capacity is made available to a database by checking the size of its data directory.

  • For PostgreSQL, the directory is /bitnami/postgresql. You can confirm its size using the command below.

    circleci-user ~ $ kubectl exec postgresql-0 -- df -h /bitnami/postgresql

Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme4n1    7.8G  404M  7.4G   3% /bitnami/postgresql

  • For MongoDB, the directory is /bitnami/mongodb.

    circleci-user ~ $ kubectl exec mongodb-0 -- df -h /bitnami/mongodb

Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme1n1    7.8G  441M  7.4G   3% /bitnami/mongodb

  • For Redis, the directory is /data.

    circleci-user ~ $ kubectl exec redis-master-0 -- df -h /data

Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme2n1     8G  156K   8G   1% /data

circleci-user ~ $ kubectl exec redis-slave-0 -- df -h /data

Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme2n1     8G  156K   8G   1% /data

From the examples above, the capacities are still 8Gi. The following steps show how to increase this to 10Gi.

2. Confirm volume expansion is allowed

Confirm that volume expansion is allowed in your cluster:

circleci-user ~ $ kubectl get sc

NAME            PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
gp2 (default)   kubernetes.io/aws-ebs   Delete          WaitForFirstConsumer   false                  1d

If your default storage class has "ALLOWVOLUMEEXPANSION" set to false, like in the above example, you can change this with the following kubectl patch command:

circleci-user ~ $ kubectl patch sc gp2 -p '{"allowVolumeExpansion": true}'

storageclass.storage.k8s.io/gp2 patched
circleci-user ~ $ kubectl get sc
NAME            PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
gp2 (default)   kubernetes.io/aws-ebs   Delete          WaitForFirstConsumer   true                  1d

Now you may proceed to expanding your volumes.

3. Delete the database’s stateful set

In this step, you will delete the stateful set, which controls your database pod. The command below deletes the referenced database’s stateful set without deleting the pod. You do not want to delete the pod itself, as this would cause downtime. In the following steps, you will redeploy your stateful set. You might chose to delete one or both stateful sets, depending on which database volumes you wish to expand. The --cascade=orphan flag is most important here.

  • PostgreSQL

    kubectl delete sts postgresql --cascade=orphan

  • MongoDB

    kubectl delete sts mongodb --cascade=orphan

  • Redis

    kubectl delete sts redis-master redis-slave --cascade=orphan

4. Update the size of the database’s PVC

Now that the stateful set has been removed, you can increase the size of our persistent volume claim to 10Gi.

  • PostgreSQL

    kubectl patch pvc data-postgresql-0 -p '{"spec": {"resources": {"requests": {"storage": "10Gi"}}}}'

  • MongoDB

    kubectl patch pvc datadir-mongodb-0 -p '{"spec": {"resources": {"requests": {"storage": "10Gi"}}}}'

  • Redis

    kubectl patch pvc redis-data-redis-master-0 -p '{"spec": {"resources": {"requests": {"storage": "10Gi"}}}}'
kubectl patch pvc redis-data-redis-slave-0 -p '{"spec": {"resources": {"requests": {"storage": "10Gi"}}}}'

5. Update helm-value file with the new PVC size

Now you need to upgrade the server installation by modifying the PVC size in the helm-value file to persist your changes. In the helm-value file, you will update the values for your PVC size to 10Gi as shown below.

  • PostgreSQL

    postgresql:
  primary:
    persistence:
      size: 10Gi

  • MongoDB

    mongodb:
  persistence:
    size: 10Gi

  • Redis

    redis:
  master:
    persistence:
      size: 10Gi
  slave:
    persistence:
      size: 10Gi

Now save and deploy your changes. This recreates the stateful set(s) that you destroyed earlier, but with the new PVC sizes, which will persist through new releases.

helm upgrade <release-name> -n <namespace> -f < helm-value-file> <chart-dictectory>

6. Validate new volume size

Once deployed, you can validate the size of the data directories assigned to our databases.

  • For PostgreSQL the directory is /bitnami/postgresql.

    circleci-user ~ $ kubectl exec postgresql-0 -- df -h /bitnami/postgresql
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme4n1    9.8G  404M  9.4G   5% /bitnami/postgresql

  • For MongoDB the directory is /bitnami/mongodb.

    circleci-user ~ $ kubectl exec mongodb-0 -- df -h /bitnami/mongodb
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme1n1    9.8G  441M  9.3G   5% /bitnami/mongodb

  • For Redis the directory is /data.

    circleci-user ~ $ kubectl exec redis-master-0 -- df -h /data
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme2n1     10G  156K   10G   1% /data

circleci-user ~ $ kubectl exec redis-slave-0 -- df -h /data
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme2n1     10G  156K   10G   1% /data

As you can see, the size of your directories has been increased.

When completing these steps, if you find, as expected, that the new pods do show the resized volumes, it is still worth checking with the kubectl describe commands shown below. In some instances the resize will fail, but the only way to know is by viewing an event in the output from kubectl describe.

  • PostgreSQL

    kubectl describe pvc data-postgresql-0

  • MongoDB

    kubectl describe pvc datadir-mongodb-0

  • Redis

    kubectl describe pvc redis-data-redis-master-0
kubectl describe pvc redis-data-redis-slave-0

A successful output looks like this:

Events:
Type    Reason                      Age   From     Message

Normal  FileSystemResizeSuccessful  19m   kubelet  MountVolume.NodeExpandVolume succeeded for volume "pvc-b3382dd7-3ecc-45b0-aeff-45edc31f48aa"

Failure might look like this:

Warning  VolumeResizeFailed  58m   volume_expand  error expanding volume "circleci-server/datadir-mongodb-0" of plugin "kubernetes.io/aws-ebs": AWS modifyVolume failed for vol-08d0861715c313887 with VolumeModificationRateExceeded: You've reached the maximum modification rate per volume limit. Wait at least 6 hours between modifications per EBS volume.
status code: 400, request id: 3bd43d1e-0420-4807-9c33-df26a4ca3f23
Normal   FileSystemResizeSuccessful  55m (x2 over 81m)  kubelet        MountVolume.NodeExpandVolume succeeded for volume "pvc-29456ce2-c7ff-492b-add4-fcf11872589f"

Troubleshoot

After following these steps, if you find that the disk size allocated to your data directories has not increased, then you may need to restart your database pods. This will cause downtime of 1-5 minutes while the databases restart. You can use the commands below to restart your databases.

  • PostgreSQL

    kubectl rollout restart sts postgresql

  • MongoDB

    kubectl rollout restart sts mongodb

  • Redis

    kubectl rollout restart sts redis-master redis-slave

Suggest an edit to this page

Make a contribution
Learn how to contribute

Still need help?

Ask the CircleCI community
Join the research community
Visit our Support site