Publishing a Python package

This tutorial covers:

1. Creating and publishing a Python package
2. Building a continuous integration pipeline
3. Automating package publishing

For many software engineers and developers, using standard libraries or built-in objects is just not enough. To save time and increase efficiency, most developers build on work done by others. Whatever the coding problem, there is likely another programmer who has already created a solution for it. There is usually no need to repeat the problem-solving process. This principle is known as Do not Repeat Yourself or DRY.

Code that goes beyond the standard library is often shared between individuals, teams, and even the entire coding community. If you have solved a coding problem, you can make it publicly available as an open source codebase for other developers to benefit from. These collections of code are called packages.

In this tutorial, we will build a simple Python package and create a continuous integration (CI) pipeline for it.

If you are using a popular programming language like JavaScript or Python there will be tools available for publishing your open source packages. For JavaScript, you can use Node Package Manager NPM. For Python, you can use the Python Package Index, also known as PyPI. We will be using PyPI for this tutorial.

Prerequisites

To complete this tutorial you will need:

1. Basic knowledge of the Python programming language
2. Python installed on your system. If you don’t have it already, you can download it
3. A CircleCI account
4. A GitHub account
5. Accounts at test.pypi.org and pypi.org

Our tutorials are platform-agnostic, but use CircleCI as an example. If you don’t have a CircleCI account, sign up for a free one here.

Note: Both test.pypi.org and pypi.org require you to verify your email address. You will not be able to publish a package without confirming. Later steps in this tutorial require that the usernames and passwords for test.pypi.org and pypi.org are similar.

Creating a Python package

As I mentioned in the introduction, if the functionality you need is not in the standard Python library, there is most likely a package out there that delivers it. There are hundreds of thousands of packages available on pypi.org. You can search for the functionality you need, or choose from packages that are trending or new.

Our first task is to create a simple conversion package that can demonstrate package functionality in a short amount of time. The package we create will allow a programmer to convert values for:

• Temperature (between Celsius and Fahrenheit)
• Distance (between kilometers and miles)
• and others, if you want to add them

Using a typical package maintenance workflow as our model, we will provide updates over time that extend the usefulness of the package and improve security. We will use a changelog to keep track of updates and inform users of the changes. More on that later.

Project structure

Choose a location in your system and create a project:

python-package
├── convsn
│   ├── __init__.py
│   └── temperature.py
├── .gitignore
└── setup.py

The project has:

• A base folder named python-package (you can use any name you prefer)
• A module/library folder named convsn (this is the actual package)
• A Python file within the package folder named temperature.py

Make sure that the module/library folder name is a unique name that is not used by an existing package in the Test Python Package Index test.pypi.org or the Python Package Index pypi.org. Make search queries to confirm the availability of the names you want to use.

With the project created and named, we can begin building functionality to convert temperature from fahrenheit to celsius and back). The logic will be contained in the file temperature.py.

Copy this code and paste it into the ./convsn/temperature.py file:

def c2f(celcius):
    return (float(celcius) * 9/5) + 32

def f2c(fahrenheit):
    return (float(fahrenheit) - 32) * 5/9

Next, copy this code into __init__.py file:

from .temperature import c2f, f2c

Copy this code into setup.py, and then update author and author_email using your information:

from setuptools import setup, find_packages

VERSION = '0.0.1'
DESCRIPTION = 'A conversion package'
LONG_DESCRIPTION = 'A package that makes it easy to convert values between several units of measurement'

setup(
    name="convsn",
    version=VERSION,
    description=DESCRIPTION,
    long_description=LONG_DESCRIPTION,
    author="<Your Name>",
    author_email="<your email>",
    license='MIT',
    packages=find_packages(),
    install_requires=[],
    keywords='conversion',
    classifiers= [
        "Development Status :: 3 - Alpha",
        "Intended Audience :: Developers",
        'License :: OSI Approved :: MIT License',
        "Programming Language :: Python :: 3",
    ]
)

To finish this step, can use the GitHub .gitignore template for Python. Copy the contents into this projects’s .gitignore file.

The code should be similar to this.

Clone it by running:

git clone -b setup --single-branch https://github.com/CIRCLECI-GWP/publish-python-package.git

Building the package files

Before we begin this step, I should mention that I’m using Python3. Depending on your Python installation, you may need to use python3 instead of python, as shown in the example.

In your terminal (at the root of the project) run:

python setup.py sdist bdist_wheel

This command creates a source distribution and a shareable wheel that can be published on pypi.org.

To test this, create a virtual Python environment.

Note: Use a different location to prevent name-clashing between modules.

Then, install the convsn package using the wheel distribution. Depending on your Python installation, you may need to use pip3.

Run:

pip install <relative-path>/python-package/dist/convsn-0.0.1-py3-none-any.whl

Create a python script file named test.py and enter:

from convsn.temperature import f2c, c2f

print(f"{f2c(32)}")  # Result should be 0.0
print(f"{c2f(0)}")  # Result should be 32.0

To test, run the script while still in the virtual Python environment.

Our Python package is ready for publishing.

Publishing the Python package

In this step of the tutorial, we will follow the workflow a real-life package creator and maintainer might use. First, we will first publish our package to test.pypi.org to make sure everything is working. When we are ready to publish to our package users, we will move on to pypi.org.

Python Packaging Authority (PYPA) have a nifty package to help with this called twine. We will install twine then publish to test.pypi.org. You will be asked to enter your credentials for the site.

In your terminal, navigate to the root folder of your package and run:

pip install twine
twine upload --repository testpypi dist/*

In a separate Python virtual environment, pip install the package. When you run pip install <package-name>, pip searches for the package files in the official Python Package Index, on pypi.org.

Run the test.py script:

pip install --index-url https://test.pypi.org/simple convsn
python3 test.py

Here is how mine executed.

Success! Now we can automate the publishing process with CircleCI.

Automating package publishing with CircleCI

It is no secret that CircleCI is great for automating scripts. It turns out it is also great for creating a repeatable process for package publication. We will create a process that can:

• Upgrade from Test PyPI to PyPI
• Maintain checks (if you include tests)
• Allow credentials to be used only by the pipeline, without sharing with every developer working on the package

To begin, create a folder named tests. In it, create a file and name it test_temperature.py. In tests/test_tempertaure.py, enter:

from convsn.temperature import f2c, c2f

def test_f2c():
    assert f2c(32) == 0.0

def test_c2f():
    assert c2f(0) == 32.0

While you are in the Python virtual environment containing our convsn package, go to the root folder. Pip install pytest, then run the pytest command:

pip install pytest
pytest

Both tests should pass successfully.

Next, add a CircleCI configuration file to the project. Create a folder named .circleci. In the new folder, create a file named config.yml. In .circleci/config.yml, copy and paste:

version: 2.1
jobs:
  build_test:
    docker:
      - image: cimg/python:3.11.0
    steps:
      - checkout # checkout source code to working directory
      - run:
          command: | # create whl and use pipenv to install dependencies
            python3 setup.py sdist bdist_wheel
            sudo add-apt-repository universe -y
            sudo apt-get update
            sudo apt install -y python3-pip
            sudo pip install pipenv
            pipenv install dist/convsn-0.0.3-py3-none-any.whl
            pipenv install pytest
      - run:
          command: | # Run test suite
            pipenv run pytest
  test_pypi_publish:
    docker:
      - image: cimg/python:3.11.0
    steps:
      - checkout # checkout source code to working directory
      - run:
          command: | # create whl, install twine and publish to Test PyPI
            python3 setup.py sdist bdist_wheel
            sudo add-apt-repository universe -y
            sudo apt-get update
            sudo apt install -y python3-pip
            sudo pip install pipenv
            pipenv install twine
            pipenv run twine upload --repository testpypi dist/*
  pypi_publish:
    docker:
      - image: cimg/python:3.11.0
    steps:
      - checkout # checkout source code to working directory
      - run:
          command: | # create whl, install twine and publish to PyPI
            python3 setup.py sdist bdist_wheel
            sudo add-apt-repository universe -y
            sudo apt-get update
            sudo apt install -y python3-pip
            sudo pip install pipenv
            pipenv install twine
            pipenv run twine upload dist/*
workflows:
  build_test_publish:
    jobs:
      - build_test
      - test_pypi_publish:
          requires:
            - build_test
          filters:
            branches:
              only:
                - develop
      - pypi_publish:
          requires:
            - build_test
          filters:
            branches:
              only:
                - main

This configuration file instructs the pipeline to install the necessary dependencies, run tests, and publish the package. The workflow part of the configuration specifies filters, the sequence the jobs should be executed in, and their dependencies.

For example, the jobs test_pypi_publish and pypi_publish cannot run if the build_test job fails. The test_pypi_publish and pypi_publish jobs run only in the develop and main branches, respectively.

Connecting the project to CircleCI

Begin by pushing your project to GitHub.

Your repository should be similar to this.

Now, log in to your CircleCI account. If you singed up with your GitHub account, all your repositories will be available on the dashboard. Click Set Up Project next to your publish-python-package project.

From the modal that appears, input the name of the branch that houses your configuration file and click Set Up Project

The build_test job will pass, but if you are in either the main or develop branches, expect the build to fail. That is because the test_pypi_publish and pypi_publish jobs cannot run yet.

Test PyPI credentials were expected when we published the package using twine, we could not interact with the terminal while it was running the command in the pipeline. To supply these credentials, we could add flags to the command: twine upload -u USERNAME -p PASSWORD. However, because the config file is tracked by Git, and our Git repository is public, using the flags would be a security risk. We can avoid this risk by creating environment variables.

Creating environment variables

While still on the CircleCI project (python-package), cancel the workflow and click Project Settings on the top right part of the page.

Select Environment Variables from the menu on the right. Then, click Add Environment Variable.

Enter the key-value pairs of TWINE_USERNAME and TWINE_PASSWORD. This is the expected result.

Our next step is creating a change log.

Adding a change log

In your local project, create a file named CHANGELOG.md and enter:

# Change Log

All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](http://keepachangelog.com/)
and this project adheres to [Semantic Versioning](http://semver.org/).

## [0.0.2] - 2022-11-11

### Added

- Temperature conversion

To make sure the build succeeds, we also need to bump the version in setup.py to 0.0.2 to match the change log.

Note: If you don’t bump the version, the publishing job will fail because you cannot publish the same version twice.

Commit the changes and push to GitHub to trigger a build.

Updating the package

To extend the package’s functionality, you can create another branch and make updates to it. After that:

1. Create tests for it
2. Bump the version in setup.py
3. Bump the version of the whl file in the CircleCI config file
4. Update the ChangeLog
5. Push to GitHub

When you have a successful build, create a GitHub Pull Request (PR) to the develop branch. If it looks good, merge it. When that build succeeds, create a PR to the main branch, review it, and merge.

Take a look at these example pull requests:

• PR to develop
• PR to main branch

The package on Test PyPI and on PyPI has the additional functionality of distance conversion from kilometer to miles and the reverse. The PRs were merged when builds were successful.

Conclusion

There you have it, a successful, repeatable process suitable for a Python package maintainer enabled by CI/CD. We went through setting up a Python package project, creating Python wheels, and installing a package using a whl file. We published the package from a local machine and then automated the process using CircleCI.

Keep building!