Mastering Python‘s pyproject.toml: The Definitive Guide

The Python packaging ecosystem continues to thrive – over 158,000 packages are available on PyPI as of January 2023. But with great success comes great complexity. Challenges abound in areas like dependency management, build configuration and environment reproducibility.

Content Navigation show

Cue pyproject.toml – an initiative to create conventions through PEP 518 and help address these pain points within the Python world.

In this comprehensive guide, we dive deep on everything pyproject.toml has to offer:

  • Its background, purpose and key motivations
  • How it enables flexibility and standardization
  • Contents, syntax and specifications
  • Packaging tutorials using various build tools
  • Best practices for optimal use

Let‘s get right to it!

Overview

First, some background. As Python packaging grew more complex over the past decade with tools like setuptools, distutils, pip and poetry – limitations around interoperability and reproducibility became apparent. Core areas that were impacted:

  • Building packages in a consistent wheel (.whl) format
  • Dependency management and environment installation
  • Metadata and configuration sharing between packaging tools

This caused challenges with reliably building code across different systems. pyproject.toml came out of the need to establish conventions and improve the status quo.

The key goals behind standardizing pyproject.toml are:

  • Flexibility: Support multiple build tools like poetry, flit, pip while outputting consistent packages
  • Interoperability: Common metadata format for tools to consume configuration and share data
  • Reproducibility: Reliably reproduce build environments across different systems
  • Simplicity: Easy to use configuration format for defining projects

This guide covers various facets of pyproject.toml in depth:

  • Configuration format, syntax and contents
  • Interoperability between build systems
  • Packaging tutorial using setuptools
  • Best practices for optimal usage

So let‘s explore the innards of this ubiquitous new standard!

What is pyproject.toml and How Does it Work?

pyproject.toml is a configuration file that contains metadata and defines how a Python project should be built and packaged.

History

Problems with Python build tools motivated the need for standardization. For example:

  • setup.py scripts became complex and error-prone
  • Dependency management through multiple files like requirements.txt
  • Lack of consistency across distribution formats like wheels, eggs, sdists

To address this, PEP 518 proposed conventions that were ratified as the pyproject.toml spec. It was first available in Python 3.6+.

Some key functionality include:

  • Declarative config using TOML syntax instead of imperative build scripts
  • Ability to support multiple build backends – setuptools, poetry, flit etc
  • Dependency declaration and environment reproducibility
  • Metadata like name, version – required by packaging systems
  • Tool configuration section for linters, formatters etc

In a nutshell, pyproject.toml handles critical aspects of Python packaging in a standard manner.

Contents

pyproject.toml uses TOML syntax – a minimal configuration language intended for clarity and readability.

It contains various sections like:

[build-system]

Defines how to build the project:

[build-system]
requires = ["setuptools", "wheel"]
build-backend = "setuptools.build_meta"

[project]

Contains metadata like name, description, authors etc:

[project]
name = "myproject"
version = "1.0"  
description = "A fun package"
authors = [ 
  { name="Alice"},
  { name="Bob" }
]

[tool.X]

Configures tools like linters and formatters: 

[tool.black]
line-length = 100

[metadata]

Additional metadata like Python version required, dependencies etc.

[metadata]  
requires-python = ">=3.6" 
dependencies = ["requests", "fastapi"]

As we can see, pyproject.toml provides a wide range of configuration – from build specification to tooling setup!

Build Systems and Interoperability

A key focus of pyproject.toml is improving interoperability between Python build systems and packaging tools. Let‘s explore this aspect.

Overview

Python has an extensive ecosystem of build tools and distribution formats including:

  • Packaging: setuptools, poetry, flit, hatch, pip
  • Formats: wheels(.whl), eggs, sdists

However, limitations around consistency meant:

  • Difficulty switching between tools like setuptools to poetry
  • Inability to reliably produce wheels across environments

pyproject.toml establishes common standards to fix this through:

  • Unified build configuration and metadata
  • Requirement to output wheels and sdists
  • Support for multiple build backends

This improves flexibility, interoperability and reproducibility.

Standardization Examples

Some examples where pyproject.toml enables standardization:

Producing wheels

The [build-system] section ensures any compliant backend can output .whl files:

[build-system]
# Build wheel packages  
build-backend = "setuptools.build_meta"

Tool configuration

Tools can read config under [tool.X]:

[tool.pylint]
max-line-length = 100 

[tool.pytest]
addopts = "-ra -q"

This allows easy toolchain integration regardless of build backend used.

Switching backends

Project metadata and dependencies are specified in a common format – allowing switching build tools:

[project]
# Name, version applies to any build tool 
name = "package"  
version = "1.0"

[metadata]
requires-python = ">=3.6"

So in summary, pyproject.toml establishes conventions that connect disparate tools to improve interoperability.

Packaging Python Projects Tutorial

Let‘s see how to leverage pyproject.toml for packaging projects using the setuptools build backend.

Project scaffolding

We will use the following folder structure:

package
├─ src
│  ├─ package 
│     ├─ __init__.py
|     ├─ main.py
├─ pyproject.toml
├─ README.md

Where:

  • src contains our Python code
  • pyproject.toml defines the build
  • README has documentation

Minimal pyproject.toml

Here is a minimal pyproject.toml:

[build-system]
requires = ["setuptools", "wheel"]  
build-backend = "setuptools.build_meta"

[project]
name = "package"
version = "1.0.0"
description = "My package"
authors = [
  { name="Alice" },
  { name="Bob" }
]

[tool.setuptools]
packages = ["package"]

We specify setuptools as the build backend along with basic metadata.
The [tool.setuptools] section tells setuptools our Python code to package is in the package folder.

Running the build

To build a wheel run:

python -m build

After the build completes, a dist folder with the *.whl package is created!

We can upload this file to PyPI for others to install our package using pip.

Additional customization

Additional things we can customize include:

  • Declaring dependencies
  • Adding package entry points
  • Supporting plugins
  • Creating platform specific wheels

And much more!

So in summary, by leveraging pyproject.toml and PEP 518 standards, we can package Python code with simplicity and consistency.

Recommendations and Best Practices

Let‘s round up the guide by going over some best practices when using pyproject.toml:

Use over setup.py

Prefer pyproject.toml over setup.py for defining builds as it‘s the newer standard.

Declare dependencies

Accurately specify dependencies in [project]/metadata sections to improve reproducibility: 

[metadata]
dependencies = [
  "requests >=2.28.1,<3",  
  "fastapi >=0.88,<0.89" 
]

Use appropriate version specifiers.

Configure tools

Make use of [tool.] sections to integrate linters, formatters etc:

[tool.isort]
line_length = 100

[tool.pyright]  
reportMissingTypeStubs = true

This avoids need for separate configuration files.

Reproducibility

Take care to freeze dependencies and isolate build environment for reproducible builds across systems.

For detailed tips, refer to the official packaging guide.

Transitioning

When migrating existing projects to pyproject.toml, reuse metadata from setup.py and requirements.txt files.

Also reference the porting guide by setuptools.

So in summary:

  • Favor pyproject.toml over older standards
  • Declare dependencies carefully with correct versions
  • Integrate tools into unified config approach
  • Take steps to improve reproducibility

This will lead to easier and more robust Python packaging!

The Road Ahead

The Python ecosystem continues to rapidly adopt pyproject.toml as the standard for project configuration – a promising trend for improving interoperability.

But active maintenance is required to balance flexibility and convention as more build tools emerge. The working group behind PEP 518 has their work cut out!

While there are still areas of debate, I believe the core focus on reproducibility and support for multiple backends means pyproject.toml is here to stay. Exciting times ahead for Python packaging!

Tags: