Continuous integration has become a widely accepted practice for software projects. As more technologies are introduced in both continuous integration and software development, developers are looking for practical ways to benefit from them. Basic tutorials that cover toy examples are not always enough for real-life practitioners. As an actual user of Django, Docker, and CircleCI, this was certainly a pain point for me. That is why I wrote this tutorial.

In this guide, you will learn from a fellow practitioner how set up a continuous integration pipeline for a production-ready Dockerized Django 3.2 application.

So that you can get started easily, I created a demo project for you to clone from GitHub.

When you have finished the tutorial, every push to your project’s GitHub repo will automatically trigger a build and upload coverage docs to a code-coverage cloud provider like Codecov.io. You will even learn how to use CircleCI’s parallel mode to run your tests faster and achieve a shorter iterative cycle.

Prerequisites

To get the most from this tutorial, you will need:

Our tutorials are platform-agnostic, but use CircleCI as an example. If you don’t have a CircleCI account, sign up for a free one here.

Because every Django project is different, I will lead you through the steps of setting up a continuous integration pipeline using the demo project.

I recommend that you follow this tutorial start-to-finish twice. The first time, use the demo project. For the second go through, use your own Django project, repeating the same steps conceptually, but tweaking the process as necessary.

I will provide some tips for tweaking the steps for your own project later on in the tutorial. For now, focus on following along with the demo project, so you develop an intuition about how the whole process works from end-to-end.

Cloning the demo project

In this step, you will git clone the codebase and make sure you are satisfied that the codebase can run the tests locally. Essentially, this codebase follows the same classic seven-part tutorial in Django’s documentation. There are some differences in the demo project codebase.

The main differences are that:

  1. The demo project maintains function-based views instead of class-based views used in the tutorial.
  2. The demo project has a config folder with a settings sub-folder that holds base.py, local.py, and production.py instead of just settings.py.
  3. The demo project has a dockerized_django_demo_circleci folder with a users sub-folder.

Git clone the codebase:

$ git clone https://github.com/CIRCLECI-GWP/dockerized-django-demo.git

Here is the output that results:

[secondary_label Output]
Cloning into 'dockerized-django-demo-circleci'...
remote: Enumerating objects: 325, done.
remote: Counting objects: 100% (325/325), done.
remote: Compressing objects: 100% (246/246), done.
remote: Total 325 (delta 83), reused 293 (delta 63), pack-reused 0
Receiving objects: 100% (325/325), 663.35 KiB | 679.00 KiB/s, done.
Resolving deltas: 100% (83/83), done.

Now that you have cloned the demo project, you are ready to test the local version of the Dockerized demo project.

Running the test locally

In this step, you will continue from the previous step to make sure tests can run successfully on your local machine.

Launch your Docker Desktop app. Refer to this tutorial if you need to review how.

Docker Desktop is ready

Go into the bash of your Docker container for the Django and run tests:

$ cd dockerized-django-demo
$ docker-compose -f local.yml run web_django bash

Following this output, enter into the bash shell of the Django Docker container.

[secondary_label Output]
Creating network "dockerized-django-demo_default" with the default driver
Creating dockerized-django-demo_db_postgres_1 ... done
Creating dockerized-django-demo_web_django_run ... done
Going to use psycopg2 to connect to postgres
psycopg2 successfully connected to postgres
PostgreSQL is available
root@abc77c0122b3:/code#

Now, run the tests.

root@abc77c0122b3:/code# python3 manage.py test --keepdb

The output will be:

[secondary_label Output]
Using existing test database for alias 'default'...
System check identified no issues (0 silenced).
..........
----------------------------------------------------------------------
Ran 10 tests in 0.454s

OK
Preserving test database for alias 'default'...

This output means that your tests are ok. This should be the same result when you push this to CircleCI.

To exit your bash shell, type exit

root@abc77c0122b3:/code# exit

Go back to your host OS.

[secondary_label Output]
exit
$

To shut down the Docker properly:

$ docker-compose -f local.yml down --remove-orphans

Here is the output:

[secondary_label Output]
Stopping dockerized-django-demo_db_postgres_1 ... done
Removing dockerized-django-demo_web_django_run_8eeefb5e1d1a ... done
Removing dockerized-django-demo_db_postgres_1               ... done
Removing network dockerized-django-demo_default

You can now safely stop Docker Desktop.

By now, your Django project and test cases should work to your satisfaction. It is now time for you to push the code to GitHub.

Connecting your own GitHub repository to CircleCI

If you have been using the demo project, and you have created your own GitHub repo, then you need to rename the remote that you cloned the code from. Add your GitHub repo as remote origin instead.

Replace all mentions of greendeploy-io/test-ddp with your repo’s actual name - org-name/repo-name

$ git remote rename origin upstream
$ git remote add origin git@github.com:greendeploy-io/test-ddp.git
$ git push -u origin main

[secondary_label Output]
Enumerating objects: 305, done.
Counting objects: 100% (305/305), done.
Delta compression using up to 10 threads
Compressing objects: 100% (222/222), done.
Writing objects: 100% (305/305), 658.94 KiB | 2.80 MiB/s, done.
Total 305 (delta 70), reused 297 (delta 67), pack-reused 0
remote: Resolving deltas: 100% (70/70), done.
To github.com:greendeploy-io/test-ddp.git
 * [new branch]      main -> main
Branch 'main' set up to track remote branch 'main' from 'origin'.

Adding the CircleCI configuration file

In this step, you will create a CircleCI configuration file and write the script to configure a continuous integration pipeline for your project. To begin, create a folder named .circleci and within it, a config.yml file. Now, open the newly created file and enter this content:

# Python CircleCI 2.0 configuration file
#
# Check https://circleci.com/docs/language-python/ for more details
#
version: 2.1

# adding dockerhub auth because dockerhub change their policy
# See https://discuss.circleci.com/t/authenticate-with-docker-to-avoid-impact-of-nov-1st-rate-limits/37567/23
# and https://support.circleci.com/hc/en-us/articles/360050623311-Docker-Hub-rate-limiting-FAQ
docker-auth: &docker-auth
  auth:
    username: $DOCKERHUB_USERNAME
    password: $DOCKERHUB_PAT

orbs:
  codecov: codecov/codecov@3.2.2

jobs:
  build:
    docker:
      # specify the version you desire here
      # use `-browsers` prefix for selenium tests, e.g. `3.7.7-browsers`
      - image: cimg/python:3.8.12
        # use YAML merge
        # https://discuss.circleci.com/t/updated-authenticate-with-docker-to-avoid-impact-of-nov-1st-rate-limits/37567/35?u=kimsia
        <<: *docker-auth
        environment:
          DATABASE_URL: postgresql://root@localhost/circle_test?sslmode=disable
          USE_DOCKER: no

      # Specify service dependencies here if necessary
      # CircleCI maintains a library of pre-built images
      # documented at https://circleci.com/docs/circleci-images/
      - image: cimg/postgres:14.1 # database image for service container available at `localhost:<port>`
        # use YAML merge
        # https://discuss.circleci.com/t/updated-authenticate-with-docker-to-avoid-impact-of-nov-1st-rate-limits/37567/35?u=kimsia
        <<: *docker-auth
        environment: # environment variables for database
          POSTGRES_USER: root
          POSTGRES_DB: circle_test

    working_directory: ~/repo

    # can check if resource is suitable at resources tab in builds
    resource_class: large

    # turn on parallelism to speed up
    parallelism: 4

    steps: # a collection of executable commands
      # add deploy key when needed esp when requirements point to github url
      # - add_ssh_keys:
      #     fingerprints:
      #       - "ab:cd:ef..."
      - checkout # special step to check out source code to the working directory
      # using dockerize to wait for dependencies
      - run:
          name: install dockerize
          command: wget https://github.com/jwilder/dockerize/releases/download/$DOCKERIZE_VERSION/dockerize-linux-amd64-$DOCKERIZE_VERSION.tar.gz && sudo tar -C /usr/local/bin -xzvf dockerize-linux-amd64-$DOCKERIZE_VERSION.tar.gz && rm dockerize-linux-amd64-$DOCKERIZE_VERSION.tar.gz
          environment:
            DOCKERIZE_VERSION: v0.4.0
      # the actual wait for database
      - run:
          name: Wait for db
          command: dockerize -wait tcp://localhost:5432 -timeout 1m
      - restore_cache: # restores saved dependency cache if the Branch key template or requirements.txt files have not changed since the previous run
          key: deps1-{{ .Branch }}-{{ checksum "requirements/base.txt" }}-{{ checksum "requirements/local.txt" }}
      - run: # install and activate virtual environment with pip
          command: |
            python3 -m venv venv
            . venv/bin/activate
            pip install --upgrade setuptools && pip install wheel
            pip install --upgrade pip==22.0.4
            pip install --upgrade pip-tools
            pip-sync requirements/base.txt requirements/local.txt
      - save_cache: # special step to save dependency cache
          key: deps1-{{ .Branch }}-{{ checksum "requirements/base.txt" }}-{{ checksum "requirements/local.txt" }}
          paths:
            - "venv"
      - run:
          name: run collectstatic
          command: |
            . venv/bin/activate
            python3 manage.py collectstatic --noinput
      - run: # run tests
          name: run tests using manage.py
          command: |
            # get test files while ignoring __init__ files
            TESTFILES=$(circleci tests glob "*/tests/*.py" | sed 's/\S\+__init__.py//g' | sed 's/\S\+factories.py//g')
            echo $TESTFILES | tr ' ' '\n' | sort | uniq > circleci_test_files.txt
            TESTFILES=$(circleci tests split --split-by=timings circleci_test_files.txt | tr "/" "." | sed 's/\.py//g')
            . venv/bin/activate
            # coverage's --parallel-mode will generate a .coverage-{random} file
            # usage: https://docs.djangoproject.com/en/3.2/topics/testing/advanced/#integration-with-coverage-py
            # add `--verbosity=3` between $TESTFILES --keepdb if need to debug
            coverage run --parallel-mode manage.py test --failfast $TESTFILES --keepdb
          # name: run tests using pytest
          # command: |
          #   . venv/bin/activate
          #   pytest -c pytest.ini -x --cov-report xml --cov-config=.coveragerc --cov
      - store_test_results:
          path: test-results
      - store_artifacts:
          path: test-results
          destination: tr1
      # save coverage file to workspace
      - persist_to_workspace:
          root: ~/repo
          paths:
            - .coverage*
  fan-in_coverage:
    docker:
      # specify the version you desire here
      # use `-browsers` prefix for selenium tests, e.g. `3.7.7-browsers`
      - image: cimg/python:3.8.12
        # use YAML merge
        # https://discuss.circleci.com/t/updated-authenticate-with-docker-to-avoid-impact-of-nov-1st-rate-limits/37567/35?u=kimsia
        <<: *docker-auth
    working_directory: ~/repo
    resource_class: small
    parallelism: 1
    steps:
      - checkout
      - attach_workspace:
          at: ~/repo
      - restore_cache: # restores saved dependency cache if the Branch key template or requirements.txt files have not changed since the previous run
          key: deps1-{{ .Branch }}-{{ checksum "requirements/base.txt" }}-{{ checksum "requirements/local.txt" }}
      - run: # install and activate virtual environment with pip
          command: |
            python3 -m venv venv
            . venv/bin/activate
            pip install --upgrade setuptools && pip install wheel
            # because of https://github.com/jazzband/pip-tools/issues/1617#issuecomment-1124289479
            pip install --upgrade pip==22.0.4
            pip install --upgrade pip-tools
            pip-sync requirements/base.txt requirements/local.txt
      - save_cache: # special step to save dependency cache
          key: deps1-{{ .Branch }}-{{ checksum "requirements/base.txt" }}-{{ checksum "requirements/local.txt" }}
          paths:
            - "venv"
      - run:
          name: combine coverage and generate XML report
          command: |
            . venv/bin/activate
            coverage combine
            # at this point, if combine succeeded, we should see a combined .coverage file
            ls -lah .coverage
            # this will generate a .coverage.xml file
            coverage xml
      - codecov/upload:
          # xtra_args: '-F'
          upload_name: "${CIRCLE_BUILD_NUM}"
workflows:
  main:
    jobs:
      - build
      - fan-in_coverage:
          requires:
            - build

To easily access Docker registry from this script, you set up a docker-auth field in a context and provided your Docker Hub credentials. These credentials will be included later in the tutorial as environment variables.

Next, the Codecov orb was pulled in from the CircleCI Orbs registry. This orb helps upload your coverage reports to Codecov without any complex configurations.

Two separate jobs build the project, run the test on CircleCI, and deploy the coverage reports to Codecov:

  • build
  • fan-in_coverage

The configured workflow will ensure that the build job is run completely before fan-in_coverage begins since its output will be uploaded to Codecov.

Job structure and workflow

As described in the previous section, there are two jobs: build and fan-in_coverage. I have collapsed them here to make it easier to review.

jobs: build:...
  fan-in_coverage:...
workflows:
  main:
    jobs:
      - build
      - fan-in_coverage:
          requires:
            - build

Notice how the fan-in_coverage requires the output from the build job and how fan-in_coverage uploads the coverage artifact to codecov.io. Using two jobs allows you to use parallelism and still allow uploading of artifacts.

Parallelism

Why use parallelism? The answer is simple: parallelism speeds up the builds.

# turn on parallelism to speed up
parallelism: 4

To allow parallelism, you need to split up the tests. To split up the tests, CircleCI needs to know the full list of test files.

Parallelism by splitting up tests

The first line tries to find all the relevant test files, skipping __init__ and factories.py. The list of files is then written into circleci_test_files.txt.

TESTFILES=$(circleci tests glob "*/tests/*.py" | sed 's/\S\+__init__.py//g' | sed 's/\S\+factories.py//g')
echo $TESTFILES | tr ' ' '\n' | sort | uniq > circleci_test_files.txt
TESTFILES=$(circleci tests split --split-by=timings circleci_test_files.txt | tr "/" "." | sed 's/\.py//g')

Then, CircleCI will run the test using coverage.py help in parallel mode. If you need to debug, add a --verbosity=3 flag.

# add `--verbosity=3` between $TESTFILES --keepdb if need to debug
coverage run --parallel-mode manage.py test --failfast $TESTFILES --keepdb

Storing artifacts and files

Finally, the config tells CircleCI where to store the artifacts and other coverage related files to be used in fan-in_coverage:

- store_test_results:
    path: test-results
- store_artifacts:
    path: test-results
    destination: tr1
# save coverage file to workspace
- persist_to_workspace:
    root: ~/repo
    paths:
      - .coverage*

Commit all the changes and update your project repository on GitHub.

Connecting your project to CircleCI

To connect your project to CircleCI, log in to your CircleCI account. If you signed up with your GitHub account, all your repositories will be available on your project’s dashboard. Locate your project from the list; in this case it is dockerized-django-demo. Click Set Up Project.

![Project settings]2022-06-06-setup-project

You will be prompted to select the .circleci/config.yml file within your project. Then, enter the name of branch and click Set Up Project.

Project settings - select config

Your workflow will start and build successfully.

Adding Docker Hub credentials to project settings as environment variables

Next, you need to add your Docker Hub username and personal access token as environment variables into the project settings.

Go to the CircleCI project page for your GitHub repo and click Project Settings. Select Environment Variables from the left menu sidebar.

Create the two environment variables: DOCKERHUB_PAT and DOCKERHUB_USERNAME. Use your Docker Hub account as a source for the values.

Environment variables

When you are done, every push to your GitHub repository or pull request should result in a build on your CircleCI.

Because the demo project is a public repo, the CircleCI project build page and Codecov coverage page are public also.

Check out the CircleCI project build page for the demo project to learn how your build page should look once you have completed all the steps.

Conclusion

In this article you cloned the demo project for a production-ready, Dockerized Django 3.2 demo project. Now you can do the same for your own Dockerized Django 3.2 project. Even more importantly, in the second half of the article, I went into deep detail about what should be in the config.yml file.

As mentioned earlier, I recommend that you follow the tutorial using the demo project the first try. The second time, when using your own project, just replace every mention of demo Django project with your Django project.

I emphasized certain parts of the explanation so that you know how they help speed up your CI/CD builds. Using parallelism speeds up builds without sacrificing key pieces like Docker and uploading to code coverage providers. One thing to note is that the set up here is for continuous delivery, not continuous deployment. Visit this page to learn more about the differences.

KimSia is an independent software engineer who writes ops software as a service to businesses. He is an avid Python developer whose projects include GreenDeploy and AutonomyFirst.app.

Read more posts by KimSia Sim