Note from the publisher: You have managed to find some of our old content and it may be outdated and/or incorrect. Try searching in our docs or on the blog for current information.

The Python Package Index commonly known as PyPI is a repository of software for the Python programming langauge. Every time you run pip install $PACKAGE you are using PyPI. In this post, you will learn how to continuously deploy your own Python packages to PyPI using git tags and CircleCI.

Over the last few weeks, I have been working on a Python wrapper for the CircleCI API. This project uses the same approach that we are going to be discussing here.

Dependencies

The only dependency that this approach requires outside of the standard Python library is the twine package.

Assumptions

This tutorial assumes the following things are true:

  1. You have an account on PyPI. If not, it is easy to get started. You can register here.
  2. You have an environment variable in your project settings called PYPI_PASSWORD that refers to your PyPI password.
  3. You have a Python package that follows standard packaging guidelines. If not, Python provides some excellent documentation on how to package and distribute projects.
  4. You are using git tags to create a release.
  5. You are using CircleCI 2.0 with workflows.

Release Workflow

A high level release workflow is described below.

  1. Once you are ready to cut a new release of your project, you update the version in setup.py and create a new git tag with git tag $VERSION.
  2. Once you push the tag to GitHub with git push --tags a new CircleCI build is triggered.
  3. You run a verification step to ensure that the git tag matches the version of my project that you added in step 1 above.
  4. CircleCI performs all of your tests (you have tests right?).
  5. Once all of your test pass, you create a new Python package and upload it to PyPI using twine.

Full Walkthrough

With all of those assumptions in place, and a high level overview in mind we are ready to dive into a real world example.

setup.py

The setup.py file is the most important part of any Python package. It provides metadata for PyPI, handles all packaging tasks, and even allows you to add custom packaging-related commands. The file from our example project is shown below:

#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
:copyright: (c) 2017 by Lev Lazinskiy
:license: MIT, see LICENSE for more details.
"""
import os
import sys

from setuptools import setup
from setuptools.command.install import install

# circleci.py version
VERSION = "1.1.1"

def readme():
    """print long description"""
    with open('README.rst') as f:
        return f.read()


class VerifyVersionCommand(install):
    """Custom command to verify that the git tag matches our version"""
    description = 'verify that the git tag matches our version'

    def run(self):
        tag = os.getenv('CIRCLE_TAG')

        if tag != VERSION:
            info = "Git tag: {0} does not match the version of this app: {1}".format(
                tag, VERSION
            )
            sys.exit(info)

setup(
    name="circleci",
    version=VERSION,
    description="Python wrapper for the CircleCI API",
    long_description=readme(),
    url="https://github.com/levlaz/circleci.py",
    author="Lev Lazinskiy",
    author_email="lev@levlaz.org",
    license="MIT",
    classifiers=[
        "Development Status :: 5 - Production/Stable",
        "Intended Audience :: Developers",
        "Intended Audience :: System Administrators",
        "License :: OSI Approved :: MIT License",
        "Operating System :: OS Independent",
        "Topic :: Software Development :: Build Tools",
        "Topic :: Software Development :: Libraries :: Python Modules",
        "Topic :: Internet",
        "Programming Language :: Python :: 3",
        "Programming Language :: Python :: 3.6",
        "Programming Language :: Python :: 3 :: Only",
    ],
    keywords='circleci ci cd api sdk',
    packages=['circleci'],
    install_requires=[
        'requests==2.18.4',
    ],
    python_requires='>=3',
    cmdclass={
        'verify': VerifyVersionCommand,
    }
)

A couple of things to note:

  1. VERSION is set as a constant at the top of the file. This should always match your most recent git tag.
  2. The setup() function is fairly standard and provides all of the necessary metadata that PyPI needs.
  3. We have a custom command called verify which will ensure that our git tag matches our VERSION. We run this command as a part of our workflow to ensure that we are releasing the proper version.

config.yml

Interesting parts of the CircleCI configuration file for this project are shown below. You can see the full file here.

Deployment Job

Once we have installed all of our project dependencies in preparation for packaging, we run the custom verify command (discussed in the previous section) to ensure that the git tag matches the version that we are about to release.

      - run:
          name: verify git tag vs. version
          command: |
            python3 -m venv venv
            . venv/bin/activate
            python setup.py verify

Next, we create a .pypirc file using the PYPI_PASSWORD environment variable that is set in our project settings.

      - run:
          name: init .pypirc
          command: |
            echo -e "[pypi]" >> ~/.pypirc
            echo -e "username = levlaz" >> ~/.pypirc
            echo -e "password = $PYPI_PASSWORD" >> ~/.pypirc

Then we create all of our packages.

      - run:
          name: create packages
          command: |
            make package

This project uses a Makefile for convenience. If you don’t want to use a Makefile you can run the commands manually in this section.

The commands to create a package are:

# create a source distribution
python setup.py sdist

# create a wheel
python setup.py bdist_wheel

Lastly, we upload the packages that we just created to PyPI using twine.

      - run:
          name: upload to pypi
          command: |
            . venv/bin/activate
            twine upload dist/*

Workflows Configuration

The deploy job that we configure is only trigged when the build is a git tag, and depends on our test job passing. The configuration for this type of setup is shown below:

workflows:
  version: 2
  build_and_deploy:
    jobs:
      - build:
          filters:
            tags:
              only: /.*/
      - deploy:
          requires:
            - build
          filters:
            tags:
              only: /[0-9]+(\.[0-9]+)*/
            branches:
              ignore: /.*/

Summary

That’s it! Now every time that you push a new git tag for your project you will automatically create a new package and upload it to PyPI. This allows you to have a completely hands-off, and reproducible continous deployment pipeline. Most importantly, by using testing, continuous integration, and continous deployment you are able to ensure that the users of your package only get the highest quality package when they run pip install $YOUR_PACKAGE.