Continuous integration for Django projects

This article focuses on setting up a Continuous Integration (CI) pipeline for a Django project. The information presented here can be used for other Python projects, also. Django is a Python framework described as the “web framework for perfectionists with deadlines”. It is a good tool for creating Minimal Viable Products (MVPs) because it is easy to set up an application and simply connect with a database using its ORM. Django is database agnostic, scales well, deploys easily, and its modular structure allows for easy test-driven development. It delivers high quality code and has excellent documentation. These features also benefit your users because it enables fast shipping of new features.

Here are the steps of this tutorial:

1. Create a Django app
2. Create tests for the app
3. Dockerize the app
4. Configure CircleCI
5. Run locally
6. Push to GitHub
7. Add a badge
8. Explore optimization with caching

Prerequisites

To follow this tutorial, you will need:

• Git
• Python
• Docker
• CircleCI CLI

Create a Django app

Django Girls offers a great tutorial on the Django framework. You will use the Django Girls tutorial to create a blog application and then set up CircleCI for it. For the database, you’ll use a flat file: .sqlite.

To get the blog app, clone this repo by typing this line into your terminal:

git clone https://github.com/CIRCLECI-GWP/django_girls_complete.git

Then, enter the directory by running:

cd django_girls_complete

After I completed the tutorial from Django Girls, I made additional changes to the application’s codebase. Since these changes are not directly related to setting up CI (Continuous Integration) for the project, I will just add links to the file changes in GitHub. Please note that the latest DjangoGirls tutorial uses Django version 4.2, however, the repository you are using has the Django version bump to version 5.1.

Below are the changes included:

• Extending the security of the app such that only authenticated users can access a form to edit/ create a post.
• Adding an .editorconfig file for maintaining consistent coding styles.
• Adding files to dockerize the app. More on this later.
• Adding ruff for Python linting and formatting.
• Updating .gitignore and add README.md and LICENSE
• Version bump to version 5.1.

To review the original, run:

git checkout original

To get the codebase after my changes, run:

git checkout main

From now on I will take you step-by-step through setting up your continuous integration pipeline, from the main branch. Run tree from your terminal to review the structure of the app:

.
├── Dockerfile
├── LICENSE
├── README.md
├── blog
│   ├── __init__.py
│   ├── admin.py
│   ├── apps.py
│   ├── forms.py
│   ├── migrations
│   │   ├── 0001_initial.py
│   │   └── __init__.py
│   ├── models.py
│   ├── static
│   │   └── css
│   │       └── blog.css
│   ├── templates
│   │   └── blog
│   │       ├── base.html
│   │       ├── icons
│   │       │   ├── file-earmark-plus.svg
│   │       │   └── pencil-fill.svg
│   │       ├── post_detail.html
│   │       ├── post_edit.html
│   │       └── post_list.html
│   ├── tests.py
│   ├── urls.py
│   └── views.py
├── docker-compose.yml
├── init.sh
├── manage.py
├── mysite
│   ├── __init__.py
│   ├── asgi.py
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py
├── requirements.txt
└── ruff.toml

9 directories, 30 files

Creating tests for the app

Your CI pipeline needs to have tests to ensure that your automated build is correct before any new commit is merged. Writing tests with Django is extensively documented here.

First, replace the code in blog/tests.py with:

from django.contrib.auth.models import User
from django.test import TestCase
from django.utils import timezone

from .models import Post
from .forms import PostForm

class PostTestCase(TestCase):
    def setUp(self):
        self.user1 = User.objects.create_user(username="admin")
        Post.objects.create(author=self.user1,
                            title="Test",
                            text="We are testing this",
                            created_date=timezone.now(),
                            published_date=timezone.now())

    def test_post_is_posted(self):
        """Posts are created"""
        post1 = Post.objects.get(title="Test")
        self.assertEqual(post1.text, "We are testing this")

    def test_valid_form_data(self):
        form = PostForm({
            'title': "Just testing",
            'text': "Repeated tests make the app foul-proof",
        })
        self.assertTrue(form.is_valid())
        post1 = form.save(commit=False)
        post1.author = self.user1
        post1.save()
        self.assertEqual(post1.title, "Just testing")
        self.assertEqual(post1.text, "Repeated tests make the app foul-proof")

    def test_blank_form_data(self):
        form = PostForm({})
        self.assertFalse(form.is_valid())
        self.assertEqual(form.errors, {
            'title': ['This field is required.'],
            'text': ['This field is required.'],
        })

What we’ve added is a PostTestCase class extending from django.test.TestCase with four methods:

• In the setUp method, defined as def setUp(self), you create one user, self.user, and a post by that user.
• The test_post_is_posted method confirms that the text of the post titled Test is We are testing this.
• The test_valid_form_data method confirms that the form saves correctly: a title and text is filled on the form to create a post, the post is saved, and its title and text confirmed to be correct.
• The test_blank_form_data method confirms that the form will throw an error when neither title nor text is filled.

Second, run the following command:

python manage.py test

Note: This command builds a test suite out of all of the test cases, extending TestCase in any file whose name begins with test and then runs that test suite.

$ python manage.py test

Found 3 test(s).
Creating test database for alias 'default'...
System check identified no issues (0 silenced).
...
----------------------------------------------------------------------
Ran 3 tests in 0.002s

OK
Destroying test database for alias 'default'...

The tests are passing! A breath of fresh air.

To start with the codebase after the tests have been added, run:

git checkout tests

Dockerize the app

Next up is dockerizing the app. What does that even mean?

To dockerize an app means to develop, deploy, and run an application in a container using Docker. This involves three key files:

• .dockerignore file: A .dockerignore file is to Docker what a .gitignore file is to Git. The files and/or folders listed therein will be ignored in the Docker context and will not be found within a Docker image. Review the .dockerignore file here.

• Dockerfile file: Defines the steps to create a Docker image. Review Dockerfile here.

• docker-compose.yml file: Whether you are running one service or multiple services, Docker compose removes the need to type out a long docker run command and allows you to run one line, docker compose up, to spin up containers in the context of the file. Review docker-compose.yml file here.

I also added an initialization script to be used when running the Dockerfile. Review init.sh file here.

CircleCI configuration

For us to integrate CircleCI into your project, you need to add a configuration file for a Python app. Create a .circleci folder in the project’s root and add a config.yml file. Copy these lines into it:

version: 2
jobs:
  build:
    docker:
      - image: circleci/python:3.6
    steps:
      - checkout
      - restore_cache:
          key: deps1-{{ .Branch }}-{{ checksum "requirements.txt" }}
      - run:
          command: |
            python3 -m venv venv
            . venv/bin/activate
            pip install -r requirements.txt
      - save_cache:
          key: deps1-{{ .Branch }}-{{ checksum "requirements.txt" }}
          paths:
            - "venv"
      - run:
          name: Running tests
          command: |
            . venv/bin/activate
            python3 manage.py test
      - store_artifacts:
          path: test-reports/
          destination: python_app

If this is your very first time doing this, you’ll notice four steps that may not be apparent:

• checkout: This command fetches your source code over SSH to the configured path (the working directory, by default).
• restore_cache: This command restores a previously saved cache.
• save_cache: This command generates and saves a cache of a file, multiple files, or folders. In this case, you save a cache of the installed Python packages obtained after running pip install ….
• store_artifacts: This command stores logs, binaries, etc. so that they are accessible by the app in later runs.

Running the CircleCI build locally

I advocate that one installs the CircleCI CLI tool to run the build locally before pushing it to GitHub and running it on CircleCI. Running the build locally keeps you from having to commit code to your online repository to confirm that the build is passing. It quickens the development cycle.

Find multiple installation options for the CircleCI CLI here. I install the CLI using Homebrew by running:

brew install circleci

Then, start by running the circleci switch command to confirm that you have the newest version. Once you have the latest version, you should have the following output:

You've already updated to the latest CLI. Please see `circleci help` for usage.

You then run circleci config validate to validate that your config file is written correctly and circleci build to build the app:

$ circleci config validate
Config file at .circleci/config.yml is valid.

$ circleci build build
Fetching latest build environment...
Docker image digest: sha256:008ba7f4223f1e26c11df9575283491b620074fa96da6961e0dcde47fb757014
.
====>> Spin up environment
.
Starting container cimg/python:3.12
.
====>> Preparing environment variables
Using build environment variables:
.
====>> Restoring cache
Error:
skipping cache - error checking storage: not supported

Step failed
====>> python3 -m venv venv
. venv/bin/activate
pip install -r requirements.txt
.
Successfully installed Django-5.1.1 asgiref-3.8.1 sqlparse-0.5.1
====>> Saving cache
Error:
Skipping cache - error checking storage: not supported

Step failed
====>> Running tests
  #!/bin/bash -eo pipefail
. venv/bin/activate
python3 manage.py test

Found 3 test(s).
Creating test database for alias 'default'...
System check identified no issues (0 silenced).
...
----------------------------------------------------------------------
Ran 3 tests in 0.002s

OK
Destroying test database for alias 'default'...
====>> Uploading artifacts
Uploading /home/circleci/project/test-reports to python_app
  No artifact files found at /home/circleci/project/test-reports
Total size uploaded: 0 B
Success!

The output from the commands concludes with Success! The local build ran successfully. With this check, you can now push the code to GitHub.

Notice that, there’s an error in the steps Restoring cache and Saving cache. Caching is not supported locally; you will explore caching at the end of this tutorial.

To start with the codebase after the CircleCI config .circleci/config.yml have been added, run:

git checkout circleci

Connecting a project to CircleCI

All you need now is to connect CircleCI to your code on GitHub and you will have the CI pipeline working like a charm, that is:

1. push code change,
2. run test,
3. merge if passing
4. REPEAT

Open GitHub in your browser and create a new repository. If you don’t have a GitHub account, you can create one here. After creating your repo, push your project to GitHub.

Then, log into CircleCI to view the dashboard. If you don’t have an account, you can sign up for a free one here. On your dashboard page, select the organization, click Projects (A in the screenshot below), and then click Set Up Project (arrow in the screenshot below) adjacent to the project name you are using. In my case, the name is django_girls_complete.

Next, specify the branch, in the modal that opens, then click Set Up Project to start the build.

A successful build will look like this:

There you have it! That’s how to set up CI for Django projects. Type this into your terminal to run the application:

docker compose up

The app will be live at <0.0.0.0:8000>.

Adding a badge

You can have a number of integrations within your codebase. Having badges in your README is a best-practice that allows you to show others how the state of these integration services stand. It help others by letting them know how the state of these services stand. To get your badge, navigate to: https://app.circleci.com/settings/project/github/<Username>/<Project>/status-badges. In my case it is: https://app.circleci.com/settings/project/github/CIRCLECI-GWP/django-girls-complete/status-badges.

Alternatively, from the project page you are on, click Project Settings.

Then, in the side menu, click Status Badges (A in the screenshot), select the branch (B in the screenshot), then Copy the embedded code (C in the screenshot).

And, finally, paste it into your README, preferably close to the top, after the brief introduction of the repository. From the README, you’ll be able to see the build status of the most recent job in the specified branch.

Exploring caching

In the screenshot showing the local CircleCI CLI run, you may have noticed the text in red: Error: Skipping cache - error checking storage: not supported. Cache is not supported when running the CircleCI build locally.

For a CircleI job (a single run of the commands in a config file) to be faster, caching is implemented. More on this here. Your setup includee steps for caching. I reran the CircleCI job to display this advantage.

The most recent job built in 10 seconds, compared to the first one at 25 seconds.

Caching is beneficial for us in overlooking the pip install -r requirements.txt command if the requirements.txt file is not modified. Of course, the first job just sets up the cache. It is no wonder that it took 15 seconds shorter. Without changing the requirements.txt file, the consecutive commits to GitHub will run in about 10 seconds.

The seconds here might seem few and insignificant but with scale, these lost seconds can turn to minutes, even hours.

Conclusion

If you have followed this tutorial, you are well on your way to mastering continuous integration for your Django projects. You started by creating a Django app and writing tests for it. Then you Dockerized the app to build it in an isolated container. Doing this is beneficial because it results in only one dependency needed to run the app on any machine: Docker. The other dependencies for the app are all installed in the Docker container. You followed some best-practices for using CircleCI:

• Running the CI build locally with the CircleCI CLI.
• Adding a badge.
• Using caching for even faster builds.