A Python Package Developer’s Cheat Sheet

Notes and thoughts on how to design and set up a clean Python package structure

Background photo by Leone Venter on Unsplash

According to developers, Python is among the top five programming languages in 2019 [1]. Based on the strength of its open-source community and high adoption levels in emerging fields such as big data, analytics, and machine learning, no one should be surprised when noticing its popularity growing in the coming years. The number of packages available for Python developers shall keep growing, too. And maybe you’ll be responsible for some of them.

When it happens, keep in mind Python is very flexible in terms of package setup… there are many docs and blog posts on this subject, by the way. But sometimes, we may get confused among so many options, mainly when getting started to package development and distribution.

The goal of this article is to describe a clean package structure, making it easier for developers to test, build, and publish it, writing as few configurations as possible while taking advantage of conventions.

A Clean Package Structure

The proposed folders and files, including testing stuff, are presented below:

project-root
├──src/
   └──package_name/
      └──__init__.py
├──tests/
   └──package_name/
├──.coveragerc
├──.gitignore
├──LICENSE
├──README.md
├──setup.cfg
└──setup.py

I’ll explore all of them but .gitignore, LICENSE, and README.md since these are widely known files. Please ask Google in case you have questions about their contents.

Let me start from setup.py, which is the package’s descriptor file. It consists of a Python script where multiple properties can be set declaratively, as shown below. The properties declared in this file are recognized by package managers such as pip and IDEs such as PyCharm, which means this is a must-have for any package.

Python package cheat sheet — initial setup file

Some properties meanings are pretty straightforward: name, version, author … but others require a bit of explanation:

• packages, package_dir: used to set where your package source files are located and the namespaces they declare. In the above example, src is configured as the sources root folder. In case it has subfolders, they’re scanned by default. Packages are only recognized if they include an __init__.py file [2];
• install_requires: a tuple with all dependencies your package needs to work — warning: add only operational dependencies; nothing related to testing or building should be put here (test dependencies are covered in the next section). You can also think of this as a partial replacement for requirements.txt in case you’re familiar with it.

These properties allow us to bundle only the files users need when working with the packages we provide them, which results in smaller distribution files. Also, they will download only the dependencies each package needs to run, avoiding unnecessary network and storage usage.

There’s a practical exercise to see it in action. The sample code for this article is available on GitHub (https://github.com/ricardolsmendes/python-package-cheat-sheet) and pip allows us to install packages hosted there. Please install Python 3.6+ and activate a virtualenv. Then:

pip install git+https://github.com/ricardolsmendes/python-package-cheat-sheet
pip freeze

You'll notice two packages were installed, as follows:

python-package-cheat-sheet==1.0.0
stringcase==1.2.0

The first one is declared in setup.py, available on the GitHub repo. The second is a required (operational) dependency for that package.

Now, let’s call the package_cheat_sheet.StringFormatter.format_to_snakecase method using the Python Interactive Shell:

python
>>> from package_cheat_sheet import StringFormatter
>>> print(StringFormatter.format_to_snakecase('FooBar'))
foo_bar
>>> exit()

As you can see, foo_bar is the output for StringFormatter.format_to_snakecase('FooBar'), which means the package installation works as expected. This a quick demonstration of how you can set up a Python package and make it available for users with a few lines of code.

Packages Also Need Automated Tests

Modern software relies on automated tests, and we can’t even think about starting the development of a Python package without them. Pytest is the most used library for this purpose, so let’s see how to integrate it into the package setup.

In the first exercise, we wore a user’s hat — now it’s time to wear a developer’s one.

First of all, please uninstall the package gathered from GitHub, clone the sample code, and reinstall it from the local sources:

pip uninstall python-package-cheat-sheet
git clone https://github.com/ricardolsmendes/python-package-cheat-sheet.git
cd python-package-cheat-sheet
pip install --editable .

The command to trigger a test suite based on the setup file is python setup.py test. It does not use pytest by default, but there’s a way to replace the default testing tool: create a setup.cfg file in the package’s root folder, setting an alias for the test command. And it’s possible to set pytest args in the same file, as follows:

[aliases]
test = pytest
[tool:pytest]
addopts = --cov --cov-report html --cov-report term-missing

• pytest dependencies are required from now on; otherwise, the command will fail after the alias is created. The dependencies will be added to setup.py, using distinct properties:

setup_requires=(
    'pytest-runner',
),
tests_require=(
    'pytest-cov',
)

• pytest-runner is responsible for adding pytest support for setup tasks
• pytest-cov will help us to generate coverage statistics for our code, as we’ll see next

One more config file must be included in the package’s root folder:

• .coveragerc controls the coverage script scope. This is pretty useful when you have folders in your project that don’t need to be monitored by the tool. In the proposed clean structure, only the src folder must be covered:

[run]
source =
    src

And we’re ready to run python setup.py test, now powered by pytest. By default, pytest looks for test files inside the tests folder. For the GitHub repo you just cloned, the expected output is:

plugins: cov-2.8.1
collected 10 items
tests/package_cheat_sheet/string_formatter_test.py ..........                                                                                                                                              [100%]
---------- coverage: platform darwin, python 3.7.4-final-0 ---------
Name                                          Stmts   Miss  Cover   
--------------------------------------------------------------------
src/package_cheat_sheet/__init__.py               2      0   100%
src/package_cheat_sheet/string_formatter.py      17      0   100%
--------------------------------------------------------------------
TOTAL                                            19      0   100%
Coverage HTML written to dir htmlcov

Please also check htmlcov/index.html after running pytest, since HTML output helps a lot when you need a deeper understanding of the coverage reports.

pytest-cov HTML output

Summary

This article presents a clean Python package structure, covering both general setup and testing instrumentation. It proposes an explicit separation of sources and test files, using Python standards, convention over configuration, and common tools to get the job done by writing as little as possible.

And that’s it!

References

• [1] Stack Overflow Developer Survey 2019: Most Popular Technologies | Programming, Scripting, and Markup Languages: https://insights.stackoverflow.com/survey/2019#technology-_-programming-scripting-and-markup-languages
• [2] Building and Distributing Packages with Setuptools, Using find_packages(): https://setuptools.readthedocs.io/en/latest/setuptools.html#using-find-packages

Changelog

• 2019–10–23: Moved pytest.ini file content to setup.cfg and removed the file.
• 2019–10–30: Added mentions to __init__.py files and the References section.