Writing tests in any programming language can be difficult, but it does not have to be. In this tutorial, I will show you how you can easily write and run tests using Flask and Pytest. I will cover these tasks:

  1. Setting up Pytest on a Flask app
  2. Writing Pytest tests
  3. Grouping tests

As a bonus, we will also integrate a CI/CD pipeline to run tests on a Flask app using CircleCI.

Prerequisites

To get the most from this tutorial, you will need:

  • Basic understanding of the Python programming language
  • Understanding of Flask framework
  • Basic understanding of testing

You will also need to have these installations and setups:

  1. Python - version >= 3.5 installed in your machine
  2. A GitHub account; you can create one here
  3. A CircleCI account; you can create one here
  4. The project repository from GitHub; clone it here

Once the project is cloned, you also need to install the dependencies using the command pip install -r requirements.txt from the root folder of the project.

Before we dive into the tutorial, it may be helpful to understand what Flask is and how it works.

Why use Flask?

Flask is a lightweight micro-web framework written in Python. Flask comes pre-installed with the core features only so that you can pick customizations depending on the requirements of your project. Flask does not make decisions about which database to use or what the default parser is. All that can be changed with modifications, which makes it extensible and flexible for the emerging needs of your development team.

In this project tutorial, we will use a simple book-retrieval API built with Flask. We will use the API to show how we write tests in a Flask application. To keep this tutorial simple, we will focus on testing the API rather than building it. The API application can be found in the api.py file in the project root directory of the cloned repository.

Now that we know what Flask is and how to use it, we can build on that knowledge by learning about Pytest and how to set it up on a Flask app.

What is Pytest and how do I use it?

Pytest is a Python testing framework designed to assist developers with writing better systems and releasing them with confidence. Small and scalable tests are simple to write with Pytest. This code snippet shows the basic layout of a Pytest test:

from api import app # Flask instance of the API

def test_index_route():
    response = app.test_client().get('/')

    assert response.status_code == 200
    assert response.data.decode('utf-8') == 'Testing, Flask!'

Testing Flask requires that we first import a Flask instance app from our api (created in our application), as seen in the previous snippet. The imported instance then exposes a test_client() method from Flask that contains the features needed to make HTTP requests to the application under test. In this case, it is to our default (/) API endpoint. We then assert that the responses received by test_client() are what we expected after decoding the byte object. In the next steps below, we’ll use this format to write our tests.

Note: By default, Pytest encodes API responses using the utf-8 codec. We need to use the decode() method to convert byte objects we receive from the test client to readable string responses, as seen in the previous code snippet.

Setting up Pytest on Flask

Installing Pytest is simple. If you cloned the repository, it is already installed, and you can skip this step. If you have not cloned the repository, run this command on the terminal:

 pip install pytest

Note: If you are using a virtual environment, activate it before doing the installation. It is best practice to use a virtual environment for isolating different applications and their associated Python packages.

Importing the Pytest module

After the installation is complete, import the Pytest module. This snippet can be placed on any test file:

import pytest

Naming conventions for tests

For this tutorial, keep your tests in a tests directory in the root folder of the application. Pytest recommends that test file names begin with the format test_*.py or end with format **_test.py, Using these formats enables Pytest to auto-discover the test files easily and also minimizes confusion when running tests. We will also precede test methods with the word test when creating tests: test_index_route():.

Writing tests with Pytest

Now that we know how to set up Pytest on a Flask app, we can start writing tests for our book-retrieval API. Our first test will be for the /bookapi/books route.

# get books data

books = [
    {
        "id": 1,
        "title": "CS50",
        "description": "Intro to CS and art of programming!",
        "author": "Havard",
        "borrowed": False
    },
    {
        "id": 2,
        "title": "Python 101",
        "description": "little python code book.",
        "author": "Will",
        "borrowed": False
    }
]

@app.route("/bookapi/books")
def get_books():
    """ function to get all books """
    return jsonify({"Books": books})

For this route, all we are doing is returning a list of books that we have hardcoded in our books variable. We will test this endpoint in the next step. Using the same format, we will define our test function and assert that the response received by the test_client() is what we expect.

import json
from api import app

def test_get_all_books():
    response = app.test_client().get('/bookapi/books')
    res = json.loads(response.data.decode('utf-8')).get("Books")
    assert type(res[0]) is dict
    assert type(res[1]) is dict
    assert res[0]['author'] == 'Havard'
    assert res[1]['author'] == 'Will'
    assert response.status_code == 200
    assert type(res) is list
    ....

This test snippet is used to make sure that we can receive a list of books. We are doing this by asserting that the response received by the test_client() is indeed our expected list of books. We are also asserting that the response contains a list of dictionaries, which are the individual books as in our defined book object. Finally, we are also asserting that the first book in the list has the author Havard and the second book has the author Will. That is all we require to write a test to fetch all books from our /bookapi/books endpoint.

We need to make sure that the test passes and to do this, we will first need to know how to run tests in Pytest.

Running tests with Pytest

To run Pytest tests, you can either use py.test or pytest in the terminal. You can also run a single file by explicitly specifying the filename after the Pytest command: pytest test_api.py. When these commands are executed, Pytest automatically finds all the tests in either the root directory, or in the specified single file.

When no test files have been explicitly defined, Pytest executes any tests in the root directory that follow the standard naming pattern, without you needing to specify filenames or directories.

Executing our first test

We have verification that our test was executed successfully! Pytest marks a passed test with a green dot .; it marks a failed test with a red F. Count the number of dots or Fs to figure out how many tests passed and failed, and in what execution order.

Note: If you are debugging the tests to the console and you need to print out a response, you can use $ pytest -s test_*.py to log to stdout. When the Pytest s option is defined, you can console messages inside the test and debug the output during the test execution.

Now that we have successfully executed our first test, we can integrate CircleCI to run our tests automatically when we push to our main branch.

Setting up Git and pushing to CircleCI

To set up CircleCI, initialize a Git repository in the project by running the command:

git init

When this is done, create a .gitignore file in the root directory. Inside the file, add any modules that should be ignored. Adding modules to this file prevents them from being added to your remote repository.

The next step is adding a commit and then pushing your project to GitHub.

Log into the CircleCI and navigate to the Projects dashboard. The GitHub repositories associated with your GitHub username or organization and the specific repository that we want to set up in CircleCI. In our case it is testing-flask-framework-with-pytest. On the Projects dashboard, select the option to set up the project. Use the option for an existing configuration and start the build.

After initiating the build, your pipeline will fail, because you need to add the customized .circleci/config.yml configuration file to GitHub. Once that is added, the project will build properly.

Setting up CircleCI

Create a .circleci directory in your root directory and then add a config.yml file there. The config file contains the CircleCI configuration for every project. We will use the CircleCI Python orb for executing our tests as part of this configuration:

version: 2.1
orbs:
  python: circleci/python@1.4.0

workflows:
  sample:
    jobs:
      - build-and-test
jobs:
  build-and-test:
    docker:
      - image: cimg/python:3.8
    steps:
      - checkout
      - python/install-packages:
          pkg-manager: pip
      - run:
          name: Run tests
          command: python -m pytest

CircleCI orbs are reusable packages of YAML code. Orbs condense multiple lines of code into a single line. In this example, that line of code is: python: circleci/python@1.2. You might need to enable organization settings to allow the use of third-party orbs in the CircleCI dashboard, or request permission from your organization’s CircleCI admin.

After setting up the configuration, you can push the configuration to GitHub. CircleCI will automatically start building your project.

Voila! From the CircleCI dashboard, click the build to review details. You can verify that running the first Pytest test and integrating it into CircleCI was a success.

Pipeline Setup Success

Now that you have successfully set up continuous integration, the next step is grouping tests and running batches of grouped tests.

Grouping and running batches of tests

As application features are added, you need to increase the number of tests to make sure that everything works. It is easy to get overwhelmed by the large number of test scripts available. You do not need to worry though, Pytest already has that figured out. Pytest allows you to run multiple tests from a single file by grouping tests.

Pytest provides markers that you use to set attributes and features for testing functions.

Using Pytest test markers

We can use markers to give tests different behaviors, like skipping tests, running a subset of tests, or even failing them. Some of the default markers bundled with Pytest include xfail, skip, and parameterize. For our project, we will create a custom marker that will group all the tests that perform GET requests to our /bookapi/books and /bookapi/book/:id endpoints.

Here is an example of the structure you will use to create a custom marker:

@pytest.mark.<markername>
def test_method():
  # test code

To use the custom marker in Pytest, define it as an argument in the pytest command:

$ pytest -m <markername>

-m <markername> is the custom marker name we will use for our tests.

It is important to note that you need to import pytest in your test file to use Pytest markers.

Markers also need to be registered so that Pytest can suppress warnings about them. Add this to your pytest.ini file:

[pytest]
markers =
    <markername>: Marker description

Using markers to group tests

For the tutorial, we want to group the tests that make the GET request to our /bookapi endpoints.

Create a customer marker called get_request and add it to the tests like this.

import pytest
...
# Other imports here

@pytest.mark.get_request
def test_get_book_by_id():
    response = app.test_client().get('/bookapi/books/1')
    res = json.loads(response.data.decode('utf-8')).get("Book")
    print(res)
    assert res['id'] == 1
    assert res['author'] == 'Havard'
    assert res['title'] == 'CS50'
    assert response.status_code == 200

Running the test with the pytest -m get_request argument executes all the tests marked with the @pytest.mark.get_request decorator in the test file. You can also verify this by running it in the terminal.

Now, commit and push your changes to GitHub and verify that the pipeline executes successfully.

Passing Tests

Excellent! All our tests were executed successfully.

Conclusion

In this tutorial, you have set up tests with Pytest, executed them, learned to group tests using Pytest markers. You learned how to use Pytest command line arguments to execute tests and how to use the test_client() method to make HTTP requests, and you know how to use the received responses in your tests. I hope you enjoyed this tutorial. If you did, share what you learned with your team. Nothing solidifies learning like teaching someone else. Until the next one, enjoy your coding projects!


Waweru Mwaura is a software engineer and a life-long learner who specializes in quality engineering. He is an author at Packt and enjoys reading about engineering, finance, and technology. You can read more about him on his web profile.