Docs
Start Building for Free
CircleCI.comAcademyBlogCommunitySupport

CircleCI server v3.x configuring external services

2 weeks ago5 min read
Server v3.x
Server Admin
On This Page

This document describes how to configure the following external services for use with a CircleCI server 3.x installation. The settings described in this guide can be found in the KOTS Admin Console. Access the KOTS Admin Console by running the following command, substituting your namespace: kubectl kots admin-console -n <YOUR_CIRCLECI_NAMESPACE>

PostgreSQL

Best practices for your PostgreSQL

Consider running at least two PostgreSQL replicas to allow recovery from primary failure and for backups. The table below shows the recommended specifications for PostgreSQL machines:

# of Daily Active Users# of PostgreSQL ReplicasCPURAMDiskNIC Speed

<50

2

8 Cores

16 GB

100 GB

1 Gbps

50 - 250

2

8 Cores

16 GB

200 GB

1 Gbps

250 - 1000

3

8 Cores

32 GB

500 GB

10 Gbps

1000 - 5000

3

8 Cores

32 GB

1 TB

10 Gbps

5000+

3

8 Cores

32 GB

1 TB

10 Gbps

Back up PostgreSQL

PostgreSQL provides official documentation for backing up and restoring your PostgreSQL 12 install, which can be found here.

We strongly recommend the following:

  • Take daily backups

  • Keep at least 30 days of backups

  • Use encrypted storage for backups as databases might contain sensitive information

  • Perform a backup before each upgrade of CircleCI server

Migrating from an internal PostgreSQL to an externalized source

When a CircleCI server instance is deployed, Postgres is deployed internally by default via its helm chart. However, as an operator, you may wish to externalize this database to have better control over scalability and availability. Once you have configured your external Postgres, you may use the guide below to migrate your Postgres data to your external database.

1. Disable the application

Disable the CircleCI server application by scaling down the application layer pods. No Data is lost in this process, but the application will be unreachable.

Scale down your application layer pods:

namespace=<your-server-namespace>
kubectl -n "$namespace" scale deploy -l "layer=application" --replicas="0"

Running kubectl -n "$namespace" get pods will show most of your pods scaling to down, leaving your database pods running including Postgres.

2. Validate access to your external PostgreSQL from within the cluster (optional)

  1. Confirm that pods within your CircleCI Server cluster can access your external Postgres. You can do this from within your internal Postgres.

    PG_POD=$(kubectl -n "$namespace" get pods | grep postgresql | tail -1 | awk '{print $1}')
kubectl exec -it -n "$namespace" "$PG_POD" -- bash

  2. While still connected to the pod run:

    psql -h <your-external-postgres-host> -U postgres -p <your-external-postgres-port>

You should be able to connect to your external Postgres at this point. If not, resolve any issues before proceeding.

3. Generate export of your internal PostgreSQL

  1. Retrieve your internal Postgres credentials:

    PG_PASSWORD=$(kubectl -n "$namespace" get secrets postgresql -o jsonpath="{.data.postgresql-password}" | base64 --decode)

  2. Connect to your Postgres pod and perform a Postgres dump:

    kubectl -n "$namespace" exec -it "$PG_POD" -- bash -c "export PGPASSWORD='$PG_PASSWORD' && pg_dumpall -U postgres -c" > circle.sql

  3. Clean up the Postgres Dump. Your internally deployed Postgres uses the username postgres. However, during the restore, the Postgres dump will drop all resources before trying to create new ones, including the postgres user. Access the Postgres pod where the dump is stored and run the following commands on the Postgres dump file to remove the lines that would delete the Postgres user.

    PG_POD=$(kubectl -n "$namespace" get pods | grep postgresql | tail -1 | awk '{print $1}')
kubectl exec -it -n "$namespace" "$PG_POD" -- bash

sed -i".bak" '/DROP ROLE postgres/d' circle.sql
sed -i".bak" '/CREATE ROLE postgres/d' circle.sql
sed -i".bak" '/ALTER ROLE postgres WITH SUPERUSER INHERIT CREATEROLE CREATEDB LOGIN REPLICATION BYPASSRLS PASSWORD/d' circle.sql

4. Restore your data in your external PostgreSQL

While still connected to your the internally deployed Postgres, restore the dumped data to your external Postgres:

psql -h <your-external-postgres-host> -U postgres -p <your-external-postgres-port> < circle.sql

Now your external Postgres will have your CircleCI server data. In the next section you will update CircleCI server to point to your external Postgres.

Connecting your external Postgres instance to CircleCI server

Screenshot of KOTS admin console config showing settings for using an external PostgreSQL instance

If you choose to use an external PostgreSQL instance, complete the following fields:

  • PostgreSQL Service Domain (required) - The domain or IP address of your PostgreSQL instance.

  • PostgreSQL Service Port (required) - The port of your PostgreSQL instance.

  • PostgreSQL Service Username (required) - A user with the appropriate privileges to access your PostgreSQL instance.

  • PostgreSQL Service Password (required) - The password of the user account used to access your PostgreSQL instance.

Once you apply your changes in KOTS admin, your CircleCI server instance will be restored with pods running as before.

MongoDB

Migrating from an internal MongoDB to an externalized Source

When a CircleCI server instance deployed, MongoDB is deployed internally by default via its helm chart. However, as an operator, you may wish to externalize this database to have better control over scalability and availability. Once you have configured your external MongoDB, you may use the guide below to migrate your Mongo data to your external database.

1. Disable the application

Disable the CircleCI server application by scaling down the application layer pods. No Data is lost in this process, but the application will be unreachable.

Scale down your application layer pods:

namespace=<your-server-namespace>
kubectl -n "$namespace" scale deploy -l "layer=application" --replicas="0"

Running kubectl -n "$namespace" get pods will show most of your pods scaling to down, leaving your database pods running, including Mongo.

2. Validate access to your external MongoDB from within the cluster (optional)

  1. Confirm that pods within your CircleCI server cluster can access your external MongoDB. You can do this from within your internal MongoDB pod:

    MONGO_POD="mongodb-0"
kubectl exec -it -n "$namespace" "$MONGO_POD" -- bash

  2. While still connected to the pod run the following:

    mongo --username <username> --password --authenticationDatabase admin --host <external-mongodb-host> --port <external-mongodb-port>

You should be able to connect to your external MongoDB at this point. If not, resolve any issues before proceeding.

3. Generate export of your internal MongoDB

  1. Retrieve your internal MongoDB credentials:

    MONGO_POD="mongodb-0"
MONGODB_USERNAME="root"
MONGODB_PASSWORD=$(kubectl -n "$namespace" get secrets mongodb -o jsonpath="{.data.mongodb-root-password}" | base64 --decode)

  2. Create a backup directory in your MongoDB pod:

    kubectl -n "$namespace" exec "$MONGO_POD" -- mkdir -p /tmp/backups/

  3. Generate a MongoDB database dump to the backup directory you just created:

    kubectl -n "$namespace" exec -it "$MONGO_POD" -- bash -c "mongodump -u '$MONGODB_USERNAME' -p '$MONGODB_PASSWORD' --authenticationDatabase admin --db=circle_ghe --out=/tmp/backups/"

4. Restore your data in your external MongoDB

Use the generated MongoDB backup to restore the data to your external MongoDB:

kubectl -n "$namespace" exec "$MONGO_POD" -- mongorestore --drop -u "$MONGODB_USERNAME" -p "$MONGODB_PASSWORD" --authenticationDatabase admin /tmp/backups/circle_ghe;

Now your external MongoDB will have your CircleCI server data. In the next section you will update CircleCI server to point to your external MongoDB.

Connecting your external MongoDB instance to CircleCI server

Screenshot of KOTS admin console config showing settings for using an external MongoDB instance

If you choose to use an external MongoDB instance, complete the following fields:

  • MongoDB connection host(s) or IP(s) (required) - The hostname or IP of your MongoDB instance. Specifying a port using a colon and multiple hosts for sharded instances are both supported.

  • Use SSL for connection to MongoDB (required) - Whether to use SSL when connecting to your external MongoDB instance.

  • Allow insecure TLS connections (required) - If you use a self-signed certificate, or one signed by a custom CA, you will need to enable this setting. However, this is an insecure setting and you should use a TLS certificate signed by a valid CA, if you can.

  • MongoDB user (required) - The username for the account to use. This account should have the dbAdmin role.

  • MongoDB password (required) - The password for the account to use.

  • MongoDB authentication source database (required) - The database that holds the account information, usually admin.

  • MongoDB authentication mechanism (required) - The authentication mechanism to use, usually SCRAM-SHA-1.

  • Additional connection options (optional) - Any other connection options you would like to use. This needs to be formatted as a query string (key=value pairs, separated by &. Special characters must be URL encoded). See the MongoDB docs for available options.

Once you apply your changes in KOTS admin, your CircleCI Server instance will be restored with pods running as before.

Vault

Screenshot of KOTS Admin Console config showing settings for using an external Vault instance
Figure 1. External Vault

If you choose to use an external Vault instance, complete the following fields:

  • URL - The URL to your Vault service.

  • Transit Path - Your Vault secrets transit path.

  • Token - The access token for vault.

Help make this document better

This guide, as well as the rest of our docs, are open source and available on GitHub. We welcome your contributions.

Need support?

Our support engineers are available to help with service issues, billing, or account related questions, and can help troubleshoot build configurations. Contact our support engineers by opening a ticket.

You can also visit our support site to find support articles, community forums, and training resources.

Creative Commons License

CircleCI Documentation by CircleCI is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.