On This Page
This document describes how to use the official CircleCI pre-built Docker container images for a database service in CircleCI.
CircleCI provides pre-built images for languages and services like databases with a lot of conveniences added into the images on CircleCI Docker Hub.
The following example shows a
.circleci/config.yml file with one job called
build. Docker is selected for the executor and the first image is the primary container where all execution occurs. This example has a second image and this will be used as the service image. The first image is the programming language Python. The Python image has
pip installed and
-browsers for browser testing. The service image gives access to additional services like databases.
PostgreSQL database testing example
In the primary image, the config defines an environment variable with the
environment key, giving it a URL. The URL tells it that it is a PostgreSQL database, so it will default to the PostgreSQL default port. This pre-built circleci image includes a database and a user already. The username is
postgres and database is
circle_test. So, you can begin with using that user and database without having to set it up yourself.
POSTGRES_USER environment variable in your CircleCI config to
postgres to add the role to the image as follows:
- image: cimg/postgres:14.0 auth: username: mydockerhub-user password: $DOCKERHUB_PASSWORD # context / project UI env-var reference environment: POSTGRES_USER: postgres
version: 2.1 jobs: build: # Primary container image where all commands run docker: - image: cimg/python:3.10 environment: TEST_DATABASE_URL: postgresql://postgres@localhost/circle_test auth: username: mydockerhub-user password: $DOCKERHUB_PASSWORD # context / project UI env-var reference # Service container image - image: cimg/postgres:14.0 environment: POSTGRES_USER: postgres auth: username: mydockerhub-user password: $DOCKERHUB_PASSWORD # context / project UI env-var reference steps: - checkout - run: sudo apt-get update - run: sudo apt-get install postgresql-client - run: whoami - run: | psql \ -d $TEST_DATABASE_URL \ -c "CREATE TABLE test (name char(25));" - run: | psql \ -d $TEST_DATABASE_URL \ -c "INSERT INTO test VALUES ('John'), ('Joanna'), ('Jennifer');"
checkout first, then install the Postgres client tools. The
cimg/postgres:14.0 image doesn’t install any client-specific database adapters. For example, for Python, you might install
psycopg2 so that you can interface with the PostgreSQL database. See Pre-Built CircleCI Services Images for the list of images.
In this example, the config installs the PostgreSQL client tools,
apt-get, to get access to
psql. Installing packages in images requires administrator privileges, therefore
sudo is used - a password is not required.
Two commands follow the
postgresql-client installation that interact with the database service. These are SQL commands that create a table called test and insert a value into that table. After committing changes and pushing them, the build is automatically triggered on CircleCI and spins up the primary container.
Note: CircleCI injects a number of convenience environment variables into the primary container that you can use in conditionals throughout the rest of your build. For example, CIRCLE_NODE_INDEX and CIRCLE_NODE_TOTAL are related to concurrent execution environments. See the Build Specific Environment Variables document for details.
When the database service spins up, it automatically creates the database
circle_test and the
postgres role that you can use to log in and run your tests. Then the database tests run to create a table and insert a value into it.
This section describes additional optional configuration for further customizing your build and avoiding race conditions.
Optimizing PostgreSQL images
cimg/postgres Docker image uses regular persistent storage on disk. Storing the database in a ramdisk may improve performance. This can be done by setting the
PGDATA: /dev/shm/pgdata/data environment variable in the service container image config.
pg_restore and similar utilities requires some extra configuration to ensure that
pg_dump invocations will also use the correct version. Add the following to your
config.yml file to enable
pg_* or equivalent database utilities:
steps: # Add the Postgres 12.0 binaries to the path. - run: echo 'export PATH=/usr/lib/postgresql/12.0/bin/:$PATH' >> $BASH_ENV
Using Dockerize to wait for dependencies
Using multiple Docker containers for your jobs may cause race conditions if the service in a container does not start before the job tries to use it. For example, your PostgreSQL container might be running, but might not be ready to accept connections. Work around this problem by using
dockerize to wait for dependencies. Following is an example of how to do this in your CircleCI
version: 2.0 jobs: build: working_directory: /your/workdir docker: - image: your/image_for_primary_container auth: username: mydockerhub-user password: $DOCKERHUB_PASSWORD # context / project UI env-var reference - image: cimg/postgres:14.0 auth: username: mydockerhub-user password: $DOCKERHUB_PASSWORD # context / project UI env-var reference environment: POSTGRES_USER: your_postgres_user POSTGRES_DB: your_postgres_test steps: - checkout - run: name: install dockerize command: wget https://github.com/jwilder/dockerize/releases/download/$DOCKERIZE_VERSION/dockerize-linux-amd64-$DOCKERIZE_VERSION.tar.gz && sudo tar -C /usr/local/bin -xzvf dockerize-linux-amd64-$DOCKERIZE_VERSION.tar.gz && rm dockerize-linux-amd64-$DOCKERIZE_VERSION.tar.gz environment: DOCKERIZE_VERSION: v0.6.1 - run: name: Wait for db command: dockerize -wait tcp://localhost:5432 -timeout 1m
It is possible to apply the same principle for the following databases:
dockerize -wait tcp://localhost:3306 -timeout 1m
dockerize -wait tcp://localhost:6379 -timeout 1m
Redis also has a CLI available:
sudo apt-get install redis-tools ; while ! redis-cli ping 2>/dev/null ; do sleep 1 ; done
- Other services such as web servers:
dockerize -wait http://localhost:80 -timeout 1m
Refer to the Database Configuration Examples document for additional configuration file examples.
Help make this document better
This guide, as well as the rest of our docs, are open source and available on GitHub. We welcome your contributions.
- Suggest an edit to this page (please read the contributing guidefirst).
- To report a problem in the documentation, or to submit feedback and comments, please open an issue on GitHub.
- CircleCI is always seeking ways to improve your experience with our platform. If you would like to share feedback, please join our research community.
Our support engineers are available to help with service issues, billing, or account related questions, and can help troubleshoot build configurations. Contact our support engineers by opening a ticket.
You can also visit our support site to find support articles, community forums, and training resources.
CircleCI Documentation by CircleCI is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.