Build

Continuous delivery for Python packages automates the release process, transforming it from a manual, error-prone series of steps into a reliable, repeatable, and traceable workflow. By leveraging GitHub Actions, you can configure your repository to automatically build and publish a new version of your package to PyPI whenever a new Git tag is pushed. This practice, often called “tag-and-release,” ensures that your published package artifacts are always built in a clean, consistent environment, eliminating the “works on my machine” problem and significantly reducing the risk of human error during deployment.

When distributing Python packages, you often need to include more than just .py files. Two common requirements are non-code data files (configuration templates, images, schemas, etc.) and binary extensions (compiled C/C++/Rust code). The packaging ecosystem handles these fundamentally different assets in distinct ways, and understanding these mechanisms is crucial for successful distribution. Including Package Data Files The setuptools library provides the primary mechanism for including non-code files. Historically, the package_data argument in setup.py was used, but modern practice favors declaring these inclusions in a pyproject.toml file using the [tool.setuptools.package-data] table. This method is more declarative and separates configuration from code.

The Importance of Versioning Versioning is the cornerstone of a healthy software ecosystem. It is the primary mechanism by which you communicate changes—and their potential impact—to your users and to automated systems that depend on your package. A well-defined versioning scheme allows users to make informed decisions about when to upgrade, helps automated tools resolve dependencies correctly, and provides a clear historical record of your project’s evolution. Without a consistent versioning strategy, you risk creating “dependency hell,” where conflicting or breaking changes in your package inadvertently break other software that relies on it. Two dominant strategies have emerged for Python and the wider software world: Semantic Versioning (SemVer) and Calendar Versioning (CalVer).

Before you upload your package to the live PyPI index, it is a critical best practice to first upload it to TestPyPI. This is a separate, isolated testing environment that mirrors the real PyPI. Using it allows you to verify that your package builds correctly, that your pyproject.toml metadata is complete and valid, and that the installation process works as intended, all without polluting the official package index with test releases or risking a broken release for your users. Once you are confident in the TestPyPI release, you can then upload the very same distribution files to the official PyPI.

The python -m build command is the modern, officially recommended tool for building Python distribution packages. It replaces the older setup.py-based approach, which directly invoked setuptools and could lead to inconsistencies. The core philosophy behind build is to provide a reliable, standardized, and isolated process for creating source distributions (sdist) and built distributions (wheel). It ensures the build environment is clean, preventing undeclared dependencies from your local development environment from inadvertently becoming part of the build process.

When distributing a Python package to the Python Package Index (PyPI), you must provide at least one built distribution format. The two primary formats are the source distribution (sdist) and the wheel (bdist_wheel). Understanding the distinction between them, their creation process, and their appropriate use is crucial for efficient and reliable distribution. A source distribution, created with python -m build --sdist, is the most basic and universal distribution format. It contains your package’s source code, setup.py or pyproject.toml, and any other files specified in your MANIFEST.in. When a user installs an sdist using pip, the package must be built on their machine. This process involves executing the setup.py script (for legacy packages) or invoking the build backend specified in pyproject.toml (for modern PEP 517/518 packages), which ultimately compiles any extension modules and places the files in the correct location for the target environment. The key advantage of an sdist is its universality; it can be installed on any platform with a compatible build toolchain. However, this is also its main drawback: the end user must have all necessary compilers (like a C compiler for packages with C extensions) and development headers (like Python.h) installed, which can be a significant barrier to installation.

The choice of build backend is one of the most consequential decisions you will make when preparing a Python package for distribution. It dictates the format of your package’s metadata, how it is built into distribution artifacts (sdist and wheel), and often influences the entire developer experience. While setuptools is the longstanding incumbent, modern alternatives like Flit, Hatch, and PDM offer streamlined workflows and declarative configuration. The backend is specified in the build-system table in your pyproject.toml file, as defined in PEP 518, which ensures that pip and other tools can install the correct dependencies before attempting to build your package.

The pyproject.toml file, introduced in PEP 518, marks a fundamental shift in the Python packaging ecosystem. It replaces the patchwork of setup.py, setup.cfg, and MANIFEST.in files with a single, static, declarative configuration file. This shift is crucial for several reasons. First, it eliminates the need to execute arbitrary code (as was required with setup.py) during package installation, significantly enhancing security and predictability. Second, being declarative, it allows tools to read and understand project metadata without needing to install the package first. Finally, it provides a standardized location for configuration of all the tools in a project’s toolchain, not just packaging.