Application logging with Flask

Without a good understanding of logs, debugging an application or looking through an error stack trace can be challenging. Luckily for Python developers, Flask logging can change the way you understand debugging and how you interact with logs produced by the application. The Flask logging module gives you a way to record errors over different severity levels. A default logging module is included in the Python standard library, and it provides both simple and advanced logging functions.

In this tutorial, I will cover how to log Flask application events based on their severity levels and how to test the logging module.

Prerequisites

To get the most from this tutorial you will need:

• Basic understanding of the Python programming language.
• Understanding of the Flask framework.
• Basic understanding of testing methods.

You also need to have the following installed:

1. Python Version >= 3.5 installed on your machine
2. A GitHub account.
3. A CircleCI account.

Cloning the sample project repository

Get started by cloning the project repository from GitHub.

Once the project is cloned, you also need to install the dependencies. Use the command pip install -r requirements.txt from the root folder of the project.

Understanding Flask logging

Flask uses the standard Python logging module to log messages: app.logger. This logger can be extended and used to log custom messages. Implementing a flexible, event logging system for Flask applications gives you the ability to know when something goes wrong while your applications are executing or when they encounter errors. The diagram below shows the different parts of the Flask logging module and how they contribute to handling application logs.

![Flask logging module]2024-03-31-flask-logging-module-flow.png{: .zoomable }

The Python logger uses four sub-modules:

• Logger is the primary interface that logs events from your application. These events, when recorded, are referred to as log records.
• Handlers direct log events/records into respective destinations.
• Formatters specify the layout of your messages when they are written by the logger.
• Filters help developers manage the log record using parameters. These parameters can be in addition to the log levels.

Implementing a Flask logger

Logging allows developers to monitor the flow of a program with actions taken. You can use loggers to track application flows like tracking transactional data in ecommerce applications or recording events when an API call interacts with a service.

To start with logging in Flask, import the logging module from Python. This logger module comes out of the box from the Python installation and does not need configuration. The Python logging module logs events based on pre-defined levels. Every recorded event becomes a log record. Each record level has a different severity level:

• Debug : 10
• Info: 20
• Warning: 30
• Error: 40
• Critical: 50

The logger records logs only when the severity is higher than their log levels. Then the logger passes them to the handlers.

This snippet shows the different types of loggers and their usage in a Flask route /.

@app.route('/')
def main():
  app.logger.debug("Debug log level")
  app.logger.info("Program running correctly")
  app.logger.warning("Warning; low disk space!")
  app.logger.error("Error!")
  app.logger.critical("Program halt!")
  return "logger levels!"

You can find this snippet in the app.py file. While the Flask application is running, go to the / home route in your browser to review the logs.

How to use the loggers:

• Debug provides developers with detailed information for diagnosing program errors.
• Info displays a confirmation message that a program’s flow behavior is executing as expected.
• Warning shows that something unexpected occurred, or that a problem might occur in the near future (low disk space, for example).
• Error indicates a serious problem, like the program failed to execute some functionality.
• Critical shows the occurrence of a serious error in the application, such as a program failure.

Configuring a basic logger

A logger that provides just the basics is enough for many applications. To configure this type of logging in your app.py file, add this:

from flask import Flask
import logging

logging.basicConfig(filename='record.log', level=logging.DEBUG)
app = Flask(__name__)

@app.route('/')
def main():
  # showing different logging levels
  app.logger.debug("debug log info")
  app.logger.info("Info log information")
  app.logger.warning("Warning log info")
  app.logger.error("Error log info")
  app.logger.critical("Critical log info")
  return "testing logging levels."

if __name__ == '__main__':
  app.run(debug=True)

This snippet specifies where Flask will log your application based on the levels from DEBUG. It also sets up the message that will be logged when you call your / home route using a client like Postman.

Note: You need to complete the logging configuration before you create the Flask app object. If the app.logger is accessed before configuration, it uses the default Python handlers.

This basic configuration using logging.basicConfig logs messages and stores the log information in a .log file. For the sample project it is the record.log file.

Now, execute your Flask application using this command:

FLASK_APP=app.py flask run

Open your client application and make a GET request to your route for the running application. In this case it is http://127.0.0.1:5000/. When the main function in your program is called, it creates the record.log file, and then it sets the logging level to DEBUG. The logging activity should appear in the file record.log and output should be something like this:

## record.log file output

DEBUG:app:debug log info
INFO:app:Info log information
WARNING:app:Warning log info
ERROR:app:Error log info
CRITICAL:app:Critical log info
INFO:werkzeug:127.0.0.1 - - [31/Mar/2024 15:18:57] "GET / HTTP/1.1" 200 -

You were able to manipulate the logger object into logging all the configured loggers based on the different logger levels. When you have different loggers set for recording different levels of information, you can disable some logs from being displayed on the console and enable others. For this configuration to print out the logs on the terminal, you can remove the file configuration filename='record.log' in the logging.basicConfig() object where logs get recorded.

While these log outputs are readable, they may not be very useful, especially because you do not know when the events occurred. To fix this, you can add a format to your logs.

Formatting log outputs

The Python formatter formats the structure of the record into specific structures that make it easy to read logs and tie them to specific events. It can be applied inside the basicConfig configuration.

The log formatter consists of the following configuration:

• %(asctime)s configures the timestamp as a string.
• %(levelname)s configures the logging level as a string.
• %(name)s configures the logger name as a string.
• %(threadName)s is the thread name.
• %(message)s configures log messages as a string.

You can apply these formatting options to your configuration for a more accurate object output:

.........
logging.basicConfig(filename='record.log',
                level=logging.DEBUG, format='%(asctime)s %(levelname)s %(name)s %(threadName)s : %(message)s')
app = Flask(__name__)
.........

Using the format configuration from the previous snippet, your log output can be tied to a specific timestamp, a specific thread, and even a particular threadname. The resulting output log makes more sense and looks cleaner when you run the application and observe your output.log file:

2024-03-31 15:18:57,627 DEBUG app Thread-5 : debug log info
2024-03-31 15:18:57,627 INFO app Thread-5 : Info log information
2024-03-31 15:18:57,627 WARNING app Thread-5 : Warning log info
2024-03-31 15:18:57,628 ERROR app Thread-5 : Error log info
2024-03-31 15:18:57,628 CRITICAL app Thread-5 : Critical log info
2024-03-31 15:18:57,628 INFO werkzeug Thread-5 : 127.0.0.1 - - [31/Mar/2024 15:18:57] "GET / HTTP/1.1" 200 -

The log levels have the messages but also a timestamp, level of the log, name of the application, the thread of the process, and finally the message of your defined logs. If you encountered an error, this makes it easy to step through the timestamp to identify the specific process, timestamp, and even message. This process is much more useful while debugging than just reading plain error logs.

Now that you have created logs with the Python basicConfig, you can write tests for your logger module.

Testing loggers

Testing Python loggers is nearly the same as writing tests for normal Python functions. To write your first test, create a file named test_app.py. Open it and add your first test snippet:

from flask import Flask
from  app import app

import logging
import unittest

class TestLogConfiguration(unittest.TestCase):
    """[config set up]
    """
    def test_INFO__level_log(self):
        """
        Verify log for INFO level
        """
        self.app = app
        self.client = self.app.test_client

        with self.assertLogs() as log:
            user_logs = self.client().get('/')
            self.assertEqual(len(log.output), 4)
            self.assertEqual(len(log.records), 4)
            self.assertIn('Info log information', log.output[0])

This test snippet uses the test_client to first make a request to the / route (just as you do while running the application). You can then verify that the log output logs the INFO level log information. The only way to know whether your tests run is by executing them. Run your tests in the terminal using this command:

pytest -s

Review the results of your run.

![Test execution]2024-03-31-test-execution.png{: .zoomable }

Congratulations! You have executed your first test successfully. Now you can extend your tests to other log levels that are defined in the application. Here’s an example:

def test_WARNING__level_log(self):
      """
      Verify log for WARNING level
      """
      self.app = app
      self.client = self.app.test_client

      with self.assertLogs() as log:
          user_logs = self.client().get('/')
          self.assertEqual(len(log.output), 4)
          self.assertIn('Warning log info', log.output[1])

def test_ERROR__level_log(self):
    """
    Verify log for ERROR level
    """
    self.app = app
    self.client = self.app.test_client

    with self.assertLogs() as log:
        user_logs = self.client().get('/')
        self.assertEqual(len(log.output), 4)
        self.assertIn('Error log info', log.output[2])

These test the log levels at each stage.

Note: The log object contains more information that can then be tested and asserted based on the needs of the application.

Your last task is to automate your tests and share the results with other members of your team. You can do so using CircleCI as your continuous integration platform.

Automating tests with CircleCI

To set up CircleCI, create a .circleci directory in your root directory, then add a config.yml file to it. The config file contains the CircleCI configuration for every project. For this set up, you will use CircleCI Python orb. Use this configuration to execute your tests:

version: 2.1
orbs:
  python: circleci/python@2.1.1

workflows:
  build-app-with-test:
    jobs:
      - build-and-test
jobs:
  build-and-test:
    docker:
      - image: cimg/python:3.9
    steps:
      - checkout
      - python/install-packages:
          pkg-manager: pip
      - python/install-packages:
          pip-dependency-file: requirements.txt
          pkg-manager: pip
      - python/install-packages:
          args: pytest
          pkg-manager: pip
          pypi-cache: false
      - run:
          name: Run tests
          command: pytest

Note: CircleCI orbs are reusable packages of YAML configurations that condense multiple lines of code into a single one: python: circleci/python@2.1.1. You may need to enable organization settings (if you are the administrator) or request permission from your organization’s CircleCI admin to allow the use of third party orbs in the CircleCI dashboard.

After setting up the configuration, add a commit and then push your project to GitHub

Now, log into the CircleCI dashboard and navigate to Projects. There will be a list of all the GitHub repositories associated with your GitHub username or organization. The specific repository for this tutorial is app-logging-flask.

![Select project]2024-03-31-pipeline-select-project.png

From the Projects dashboard, select the option to set up the selected project and choose the option to use the .circleci/config.yml file and enter the branch where your config file is housed. Click Set Up Project to proceed.

This will trigger the pipeline and build successfully. Check the CircleCI dashboard and expand the build details to verify that you have run your first PyTest test successfully. It should be integrated into CircleCI.

![Pipeline step success]2024-03-31-pipeline-step-success.png

Excellent!

Conclusion

In this tutorial you have learned how to configure an API for different log output levels and how to format log output in a way that is not only straightforward but makes it easy to identify issues when they occur. Plus, you learned to write tests for logger methods and to assert different messages that have been defined to be logged.

That brings us to the end of this tutorial! Not only have you written Flask loggers, you have also tested them, and shared them with the world – or at least with the rest of your team.