Testing a Flask framework with Pytest

Writing tests in any programming language can be difficult, but it does not have to be. In this tutorial, I will show you how you can easily write and run tests using Flask and Pytest.

As a bonus, we will also integrate these Flask tests into a CI/CD pipeline using CircleCI.

Be sure to check out our other Flask tutorials to learn about application logging, authentication decorators, and automating Flask deployments. You can also learn more about Pytest in Pytest: Getting started with automated testing for Python.

Prerequisites

To get the most from this tutorial, you will need:

• Basic understanding of the Python programming language
• Understanding of Flask framework
• Basic understanding of software testing

You will also need to have these installations and setups:

1. Python - version >= 3.5 installed in your machine
2. A GitHub account
3. A CircleCI account
4. A clone of the sample project

Once the project is cloned, you also need to install the dependencies using the command pip install -r requirements.txt from the root folder of the project.

Before you get too far into the tutorial, it may be helpful to understand what Flask is and how it works.

Why use Flask?

Flask is a lightweight web framework written in Python. Flask comes preinstalled with only the core features so that you can pick customizations depending on the requirements of your project. Flask does not make decisions about which database to use or what the default parser is. All that can be changed with modifications, which makes it extensible and flexible for the emerging needs of your development team.

In this project tutorial, we will use a simple book-retrieval API built with Flask. We will use the API to show how we write tests in a Flask application. To keep this tutorial simple, we will focus on testing the API rather than building it. The API application can be found in the api.py file in the project root directory of the cloned repository.

Now that we know what Flask is and how to use it, we can build on that knowledge by learning about Pytest and how to set it up on a Flask app.

What is Pytest and how do I use it?

Pytest is a Python testing framework designed to assist developers with writing better systems and releasing them with confidence. Small and scalable tests are simple to write with Pytest. This code snippet shows the basic layout of a Pytest test:

from api import app # Flask instance of the API

def test_index_route():
    response = app.test_client().get('/')

    assert response.status_code == 200
    assert response.data.decode('utf-8') == 'Testing, Flask!'

As shown in this snippet, testing Flask requires that we first import a Flask instance app from our api (created in our application). The imported instance then exposes a test_client() method from Flask that contains the features needed to make HTTP requests to the application under test. In this case, it is to our default (/) API endpoint. We then assert that the responses received by test_client() are what we expected after decoding the byte object. In the next steps below, we’ll use this format to write our tests.

Note: By default, Pytest encodes API responses using the utf-8 codec. We need to use the decode() method to convert byte objects we receive from the test client to readable string responses.

Setting up Pytest for Flask

Installing Pytest is simple. If you cloned the repository, it is already installed, and you can skip this step. If you have not cloned the repository, run this command on the terminal for your project:

pip install pytest

Note: If you are using a virtual environment, you will want to activate it before doing the installation. It is best practice to use a virtual environment for isolating different applications and their associated Python packages.

Importing the Pytest module

After the installation is complete, import the Pytest module. This snippet can be placed on any test file:

import pytest

Naming conventions for tests

For this tutorial, keep your tests in a tests directory in the root folder of the application. Pytest recommends that test file names begin with the format test_*.py or end with format **_test.py, Using these formats enables Pytest to auto-discover the test files easily and also minimizes confusion when running tests. We will also precede test methods with the word test when creating tests: test_index_route():.

Writing tests with Pytest

Now that we know how to set up Pytest on a Flask app, we can start writing tests for our book-retrieval API. Our first test will be for the /bookapi/books route. The definition of this route is below:

# get books data

books = [
    {
        "id": 1,
        "title": "CS50",
        "description": "Intro to CS and art of programming!",
        "author": "Havard",
        "borrowed": False
    },
    {
        "id": 2,
        "title": "Python 101",
        "description": "little python code book.",
        "author": "Will",
        "borrowed": False
    }
]

@app.route("/bookapi/books")
def get_books():
    """ function to get all books """
    return jsonify({"Books": books})

For this route, all we are doing is returning a list of books that we have hardcoded in our books variable. We will test this endpoint in the next step. Using the same format, we will define our test function and assert that the response received by the test_client() is what we expect.

import json
from api import app

def test_get_all_books():
    response = app.test_client().get('/bookapi/books')
    res = json.loads(response.data.decode('utf-8')).get("Books")
    assert type(res[0]) is dict
    assert type(res[1]) is dict
    assert res[0]['author'] == 'Havard'
    assert res[1]['author'] == 'Will'
    assert response.status_code == 200
    assert type(res) is list
    ....

This test snippet is used to make sure that we can receive a list of books. We are doing this by asserting that the response received by the test_client() is indeed our expected list of books. We are also asserting that the response contains a list of dictionaries, which are the individual books as in our defined book object. Finally, we are also asserting that the first book in the list has the author Havard and the second book has the author Will. That is all we require to write a test to fetch all books from our /bookapi/books endpoint.

Next, we’ll go over how to run test in Pytest to make sure these tests are passing.

Running tests with Pytest

To run Pytest tests, you can either use py.test or pytest in the terminal. You can also run a single file by explicitly specifying the filename after the Pytest command: pytest test_api.py. When these commands are executed, Pytest automatically finds all the tests in either the root directory, or in the specified single file.

When no test files have been explicitly defined, Pytest executes any tests in the root directory that follow the standard naming pattern, without you needing to specify filenames or directories.

We have verification that our test was executed successfully! Pytest marks a passed test with a green dot .; it marks a failed test with a red F. Count the number of dots or Fs to figure out how many tests passed and failed, and in what execution order.

Note: If you are debugging the tests to the console and you need to print out a response, you can use $ pytest -s test_*.py to log to stdout. When the Pytest s option is defined, you can console messages inside the test and debug the output during the test execution.

Now that we have successfully executed our first test, we can integrate CircleCI to run our tests automatically when we push to our main branch.

Setting up CircleCI

Create a .circleci directory in your root directory and then add a config.yml file there. The config file contains the CircleCI configuration for every project. We will use the CircleCI Python orb for executing our tests as part of this configuration:

version: 2.1
orbs:
  python: circleci/python@2.1.1
jobs:
  build-and-test:
    docker:
      - image: cimg/python:3.11.0
    steps:
      - checkout
      - python/install-packages:
          pkg-manager: pip
      - run:
          name: Run tests
          command: pytest
workflows:
  sample:
    jobs:
      - build-and-test

CircleCI orbs are reusable packages of YAML code. Orbs condense multiple lines of code into a single line. In this example, that line of code is: python: circleci/python@2.1.1. You might need to enable organization settings to allow the use of third-party orbs in the CircleCI dashboard, or request permission from your organization’s CircleCI admin.

After setting up the configuration, commit your changes and then push your changes to GitHub. If you’re trying to push the sample project to GitHub, you’ll need to set your origin-url to your own repository in GitHub. To do that, create a new project on Github. Then run:

git remote set-url origin <your-repo-url>

Next, log into CircleCI and navigate to your organization home. Select Create Project, then Github.

Select your project from the dropdown, give it a meaningful name, then click the Create Project button.

Next, click Set Up Project to start building your project on CircleCI. This will display a modal popup with a default option to use the configuration file within your project’s repository. Enter the name of the branch where the configuration file is housed:

You’ll be taken to your project homepage, where you’ll see that your pipeline hasn’t run yet. To trigger your first pipeline run, push a small, meaningless commit.

Voila! From the CircleCI dashboard, click the build to review details. You can verify that running the first Pytest test and integrating it into CircleCI was a success.

Now that you have successfully set up continuous integration, the next step is grouping tests and running batches of grouped tests.

Grouping and running batches of tests

As application features are added, you need to increase the number of tests to make sure that everything works. It is easy to get overwhelmed by the large number of test scripts available. You do not need to worry though, Pytest already has that figured out. Pytest allows you to run multiple tests from a single file by grouping tests.

Pytest provides markers that you use to set attributes and features for testing functions.

Using Pytest test markers

We can use markers to give tests different behaviors, like skipping tests, running a subset of tests, or even failing them. Some of the default markers bundled with Pytest include xfail, skip, and parameterize. For our project, we will create a custom marker that will group all the tests that perform GET requests to our /bookapi/books and /bookapi/book/:id endpoints.

Here is an example of the structure you will use to create a custom marker:

@pytest.mark.<markername>
def test_method():
  # test code

To use the custom marker in Pytest, define it as an argument in the pytest command:

$ pytest -m <markername>

-m <markername> is the custom marker name we will use for our tests.

It is important to note that you need to import pytest in your test file to use Pytest markers.

Markers also need to be registered so that Pytest can suppress warnings about them. Add this to your pytest.ini file:

[pytest]
markers =
    <markername>: Marker description

Using markers to group tests

For the tutorial, we want to group the tests that make the GET request to our /bookapi endpoints.

Create a marker called get_request and add it to the tests like this.

import pytest
...
# Other imports here

@pytest.mark.get_request
def test_get_book_by_id():
    response = app.test_client().get('/bookapi/books/1')
    res = json.loads(response.data.decode('utf-8')).get("Book")
    print(res)
    assert res['id'] == 1
    assert res['author'] == 'Havard'
    assert res['title'] == 'CS50'
    assert response.status_code == 200

Running the test with the pytest -m get_request argument executes all the tests marked with the @pytest.mark.get_request decorator in the test file. You can also verify this by running it in the terminal.

Now, commit and push your changes to GitHub and verify that the pipeline executes successfully.

Excellent! All our tests were executed successfully.

Conclusion

In this tutorial, you have set up tests with Pytest, executed them, and learned to group tests using Pytest markers. You learned how to use Pytest command line arguments to execute tests and how to use the test_client() method to make HTTP requests, and you know how to use the received responses in your tests. Finally, you set up a continuous integration pipeline with CircleCI to automate your tests and and ensure consistent validation of your code changes.

I hope you enjoyed this tutorial. If you did, share what you learned with your team. Nothing solidifies learning like teaching someone else. Until the next one, enjoy your coding projects!