---
title: Python packaging with pyproject.toml and setuptools
date: 2023-11-04
tags: [code, python]
description: Python packaging has been in a bad state for ages. In this post I am going to explain how I do package management without loosing my mind.
---

Python packaging has been in a bad state for ages. I recently read a [post by
Gregory
Szorc](https://gregoryszorc.com/blog/2023/10/30/my-user-experience-porting-off-setup.py/)
that resonated with me a lot. Still, I do not personally have any issues
in practice. So in this post I am going to explain how I do package management
without loosing my mind.

## Be aware of the different kinds of tools in package management

Package management in python is highly modular, and each part of the process
can have multiple implementations. In this article I will use `setuptools` as a
build backend (the part that actually builds the package), `build` as a build
frontend (the part that creates a build environment) and `pip` to install
packages. But I could also use `poetry` to cover all of those roles with a
single tool.

There are standards that allow all of these tools to work together.
[PEP 427](https://peps.python.org/pep-0427/) defines the "wheel" package
format. [PEP 517](https://peps.python.org/pep-0517/) defines the interface
between build backends and frontends.

Note that the separation is not always razor sharp. For example, a build
frontend may also have to install packages into the build environment. And an
installer might have to act as a build frontend if the package is not available
as a wheel.

For an excellent overview of the different tools and options for package
management, see [this post by Anna-Lena
Popkes](https://alpopkes.com/posts/python/packaging_tools/).

## Don't create a package if you want an environment

Many modern package managers like npm, cargo, or Poetry automatically create
lockfiles and recommend to commit them to version control. I really don't
understand why they are doing this. A package is supposed to be installed along
with other packages, so it needs to be compatible with as many versions as
possible. I do understand that you sometimes want a reproducible environment.
But those are two separate things.

If you want to create a reproducible environment, you can use a simple
[requirements.txt
file](https://pip.pypa.io/en/stable/reference/requirements-file-format/) and
install it with `python -m pip install -r requirements.txt`. The file could
look like this:

```
# allow a range of versions
foo >= 1.1, < 2.0

# select optional features
bar[feature]

# pin a specific version and specifiy a hash for supply chain integrity
baz == 1.2.3 --hash=sha256:f22fa1e554c9ddfd16e6e41ac79759e17be9e492b3587efa038054674760e72d
```

There are some tools that can help you generate these files, e.g. `pip freeze`
or [pip-tools](https://github.com/jazzband/pip-tools). Still, I find that you
should not have too much automation in this area. The goal is that you have
control over the environment, not the other way around.

## Use venv instead of virtualenv

You usually want to install the requirements for each project into a separate
environment. That approach was pioneered by the package
[`virtualenv`](https://virtualenv.pypa.io/en/latest/). However, the
functionality was so useful that it was integrated into the standard library in
python 3.3 (2012). I still see references to `virtualenv` more than 10 years
later, but you really don't need it. Just run `python -m venv` instead.

## Use pyproject.toml to specify package meta data

I stuck with `setup.py` and `setup.cfg` pretty long. The most important reason
was that editable installs were not supported when using pyproject.toml.
Fortunately, this is not a problem because setuptools can still read meta data
from those files.

But the future is pyproject.toml and both setuptools (>= 64) and pip (>= 21.3)
now support editable installs. Note that [Ubuntu 22.04 is still on setuptools
59](https://packages.ubuntu.com/jammy/python3-setuptools). I have started
porting some projects to the new system. But I will probably wait with some
more critical projects until the new features are widely available.

The [setuptools
documentation](https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html)
on pyproject.toml is solid and porting an existing `setup.py` or `setup.cfg` to
the new syntax should be simple enough. You can also use
[`ini2toml`](https://github.com/abravalheri/ini2toml) to automatically do the
conversion.

Once you have created that file you can build your package either by using
`python -m build` (which I see recommended in most places) or `python -m pip
wheel .` (which doesn't require an additional tool). To upload your package to
PyPI you can use [twine](https://twine.readthedocs.io/en/stable/).

## Include data files

setuptools will automatically include python files in the package. If you need
to include other files, e.g. templates or translations, you traditionally had
to use a separate `MANIFEST.in` file. That still works, but it can also be
included in pyproject.toml directly:

```
[tool.setuptools.package-data]
mypackage = [
    "**/*.html",
    "**/*.csv",
]
```

## Use backend-specific configuration for more complex packages

So far we discussed pure python packages. Packages that contain C code or
similar are much more complicated for several reasons. First because they need
an additional compile step, and second because we need to build different
binary packages for different architectures.

For setuptools you [still configure that in
`setup.py`](https://setuptools.pypa.io/en/latest/userguide/ext_modules.html).
(`setup.py` is not deprecated. It is just no longer necessary for simple
packages.) However, there are other, more specialized build backends like
[scikit-build-core](https://github.com/scikit-build/scikit-build-core) or
[meson-python](https://meson-python.readthedocs.io/en/latest/tutorials/introduction.html).

## Configure other tools

Most tools can be configured using pyproject.toml, e.g.
[pytest](https://docs.pytest.org/en/7.1.x/reference/customize.html#pyproject-toml),
[coverage](https://coverage.readthedocs.io/en/latest/config.html), or
[isort](https://pycqa.github.io/isort/docs/configuration/config_files.html#pyprojecttoml-preferred-format).
A prominent exception is
[flake8](https://github.com/PyCQA/flake8/issues/234#issuecomment-812800722),
but you can replace most of it by
[ruff](https://docs.astral.sh/ruff/configuration/#using-pyprojecttoml).

## Conclusion

Python's packaging infrastructure is certainly not it's best feature, but it is
still usable. The transition to pyproject.toml took far too long and was far
too messy, but I am confident that we will finally be done with it in just a
few years.

In this article I stuck with setuptools, because that is the build backend I
know best. However, setuptools has accumulated a lot of legacy code over the
years. [flit](https://flit.pypa.io/en/latest/rationale.html) is another backend
that has "not being setuptools" as its main feature.

I wouldn't say that python packaging is good now. But at least it has
stabilized to a degree that I feel like we could actually, finally reap the
benefits of blowing up everything.