Package template (setup.py) for Cython projects

This projects provides a setuptools-based setup.py template for Cython projects with git-based automated versioning, and some notes on packaging.

This is a fork of Technologicat's original version, somewhat adapted to my needs.

TL;DR

Copy setup.py, mylibrary/_version.py and mylibrary/_static_version.py, customize, enjoy.

Introduction

Setuptools[1] has become the tool of choice[2] for packaging Python projects, yet not much documentation is available on how to use setuptools in Cython projects. As of this writing (Cython 0.27), Cython's official packaging instructions are mostly based on distutils.

For packages to be distributed (especially through PyPI), setuptools is preferable, since the documentation on distributing packages[3] assumes that is what the developer uses. Also, setuptools adds dependency resolution (over distutils), which is an essential feature of pip.

This very minimal project documents the author's best guess at current best practices for the packaging and distribution of Cython projects, by piecing together information from various sources. Possible corrections, if any, are welcome.

General notes

A (very short) terminology can be found in the Python documentation on distributing Python modules. Probably the best practical documentation on actually distributing your own Python projects, though, is the packaging guide.

There is a timeline of the history of Python packaging (as of August 2017, up to 2015) on the PyPA website.

This blog post, dated 2012 (before the introduction of the wheel format), explains many interesting technical details, such as the differences between distutils, setuptools and early pip, the install directory structure used by setuptools, .pth files, and the two kinds of .eggs, namely directories and zipfiles.

If you are familiar with distutils, but new to setuptools, see the list of new and changed keywords in the setuptools documentation.

Content and aim

The included setup.py is a setuptools-based packaging and install script template for new Cython projects.

This is similar to simple-cython-example, but our focus is on numerical scientific projects, where a custom Cython extension (containing all-new code) can bring a large speedup. The aim is to help open-sourcing such extensions in a manner that lets others effortlessly compile them, thus advancing the openness and repeatability of science. Originally based on Technologicat's original setup-template-cython, this particular version contains several modifications, supports automatic (git-based) versioning, and offers better support for Mac OS-X.

For completeness, a minimal Cython-based example library is included, containing examples of things such as absolute cimports, subpackages, NumPyDoc style docstrings, and using memoryviews for passing arrays (for the last two, see compute.pyx). The example in the test/ subdirectory demonstrates usage of the example library after it is installed.

A pruned-down version of setup.py for pure Python projects, called setup-purepython.py, is also provided for comparison.

Features

Our setup.py features the following:

• A sample project template
• The most important fields of setup()
  • If this is all you need, simple-cython-example is much cleaner.
  • If this is all you need, and you somehow ended up here even though your project is pure Python, PyPA's sampleproject (as mentioned in [4]) has more detail on this.
• Get absolute_import working in a Cython project
  • For compatibility with both Python 3 and Python 2.7 (with from __future__ import absolute_import)
  • For scientific developers used to Python 2.7, this is perhaps the only tricky part in getting custom Cython code to play nicely with Python 3. (As noted elsewhere[a][b], it is time to move to Python 3.)
• Automatically grab __version__ from git tags, so that you DontRepeatYourself declaring your package version (based on [5])
• Support for compiler and linker flags for OpenMP, to support cython.parallel.prange.
• Make setup.py pick up non-package data files, such as your documentation and usage examples (based on [6]). However, see the section on Packaging data files below.
• Make setup.py pick up data files inside your Python packages.
• Enforce that setup.py is running under a given minimum Python version (considered harmful, but if duck-checking for individual features is not an option for a reason or another) (based on [7]).
• Disable zip_safe: Having zip_safe enabled (which will in practice happen by default) is a bad idea for Cython projects, because:
  • Cython (as of this writing, version 0.25) will not see .pxd headers inside installed .egg zipfiles. Thus other libraries cannot cimport modules from yours if it has zip_safe set.
  • At (Python-level) import time, the OS's dynamic library loader usually needs to have the .so unzipped (from the .egg zipfile) to a temporary directory anyway.

In contrast to Technologicat's original version](https://github.com/Technologicat/setup-template-cython), I modified the following:

• The mechanism of using Cython in cooperation with setuptools was slightly changed in order to support target systems not having Cython installed;
• Added an improved clean command;
• Added a cython command to manually regenerating C source files for cython-based extensions;
• Changed getting the version information from a hardcoded value in the projects __init__.py (based on [8]) to using miniver
• Removed hardcoded compiler and linker switches, including support for debug vs. optimised builds;
• Removed the distinction math vs. non-math extensions, instead always linking with -lm.

Usage

See the setuptools manual. Perhaps the most useful commands are:

python setup.py build_ext  # compile binary (Cython) extensions
python setup.py build      # copy .py files in the Python packages into the build directory
python setup.py install    # will automatically "build" and "bdist" first
python setup.py sdist      # create source distribution

Substitute python2 or python3 for python if needed.

For build_ext, the switch --inplace may be useful for one-file throwaway projects, but packages to be installed are generally much better off by letting setuptools create a build/ subdirectory.

For install, the switch --user may be useful. As can, alternatively, running the command through sudo, depending on your installation.

Uninstalling your installed package

Sometimes it is useful to uninstall the installed copy of your package, such as during the development and testing of the install step for a new version.

Whereas setuptools does not know how to uninstall packages, pip does. This applies also to setuptools-based packages not installed by pip itself.

(In contrast, legacy distutils-based packages contain no metadata and cannot be automatically uninstalled.)

As an example, to uninstall this template project (if you have installed it):

pip uninstall setup-template-cython

The package name is the name argument provided to setup() in setup.py.

Note that, when you invoke the command, if the current working directory of your terminal has a subdirectory with the same name as the package to be uninstalled (here setup-template-cython), its presence will mask the package, which is probably not what you want.

If you have installed several versions of the package manually, the above command will uninstall only the most recent version. In this case, invoke the command several times until it reports that setup-template-cython is not installed.

Note that pip will automatically check also the user directory of the current user (packages installed with python setup.py install --user) for the package to uninstall; there is no need to specify any options for that.

Substitute pip2 or pip3 for pip as needed; run through sudo if needed.

To check whether your default pip manages your Python 2 or Python 3 packages, use pip --version.

Distributing binaries

Windows and Mac OS users may be interested in python setup.py bdist_wheel to create platform wheels (platform-specific binary distributions).

As for Linux, as noted in the Python packaging guide, PyPI accepts platform wheels for Linux only if they conform to the manylinux1 ABI, so in this case running python setup.py bdist_wheel on an arbitrary development machine is generally not very useful for the purposes of distribution.

For the adventurous, PyPA provides instructions along with a Docker image.

For the less adventurous, just make an sdist and upload that; scientific Linux users are likely not scared by an automatic compilation step, and will have the development tools already installed anyway.

Packaging your data files

This is rather intricate. From the viewpoint of Python packaging, data files in your project come in two varieties:

• Non-package data files are files in the project that are to be distributed, but do not belong to any Python package. Typically, this means README.md et al., end-user documentation, and usage examples.
• Package data are data files inside Python packages. Typically, this means data files needed by your library (to be loaded at runtime via the pkg_resources API, TL;DR [1] [2]), or developer documentation specific to a particular package that you want to install into the same location as the package itself.

Non-package data files arguably have no natural install location, unless they are specified with an absolute target path. Thus it is almost always better to package them only into the source distribution (sdist).

On what gets included into the sdist by default, refer to the documentation. GitHub users specifically note that README.txt gets included, but README.md does not.

Mechanisms

There are three main mechanisms to make setuptools pick up data files: data_files, package_data, and the manifest template MANIFEST.in.

The data_files option of setup() is meant for non-package data files. However, any data_files specified with a relative path will install directly under sys.prefix. Importantly, Python environments in different operating systems may behave differently.

For example, on Linux Mint, when setuptools installs packages, each .egg directory effectively (if not strictly speaking) becomes the prefix for that particular package (as far as the install procedure is concerned).

However, on Mac OS, setuptools will use the system prefix, typically /usr/local. If setup.py specifies (for example) test/* to be included as data_files, then these files will try to install into /usr/local/test/*, which will fail (for good reason).

Thus, although setup.py provides an example of this, using data_files is usually not recommended.

For a long time, the package_data option of setup() was used only for binary distributions and installation, and was ignored for the sdist [1] [2]. The manifest template MANIFEST.in was the way to customize the sdist.

However, both of these features have since been partially extended to cover some tasks of the other, perhaps in an attempt to simplify packaging in simple cases.

The documentation says that in Python 3.1+ (and also in 2.7), all files specified as package_data will be included also into the sdist, but only if no manifest template is provided.

The manifest template MANIFEST.in is an optional, separate configuration file for setuptools, to be placed in the same directory as setup.py. It can be used to include additional files (both package and non-package data) into the sdist, and to exclude any undesired files that would be included in the sdist by default.

Nowadays, files listed in MANIFEST.in can also be included in binary distributions and installation, by setting include_package_data=True in the call to setup(). The option has no effect on the contents of the sdist. Also, as the name suggests, it only affects files that reside inside Python packages; any non-package data files included by MANIFEST.in will still be packaged only into the sdist.

Format of MANIFEST.in

For an overview, see this quick explanation. For available commands, see the (very short) documentation.

Simple example MANIFEST.in:

include *.md
include doc/*.txt
exclude test/testing_scratchpad.py

In this example, the argument on each line is a shellglob. Relative paths start from the directory where setup.py and MANIFEST.in are located.

Summary

data_files:

• Meant for non-package data files.
• Not recommended. Relative paths for install targets (output) may cause installation to fail, depending on the configuration of the user's Python environment.
• May have some limited use, if an absolute target path for installation is applicable. However, note that some developers frown upon (ab)using pip for appinit (see last part here).
• Relative paths for files to include (input) are specified as relative to the directory setup.py resides in.

package_data:

• Package data only.
• Historically, main way to control binary distributions and installation.
• Now package_data is also included into sdist, if you don't provide a MANIFEST.in.
• Paths to the data files are specified as relative to each package in question. See the example in setup.py.

MANIFEST.in:

• Both package data and non-package data files.
• Historically, main way to control sdist.
• Now also controls binary distributions and installation (of any package data included by it), if you set include_package_data=True in your call to setup(). On the sdist, the option has no effect.
• Optional. If not provided, the sdist will include certain files by default.
• If this file exists, package_data will be ignored for sdist; only MANIFEST.in will be used to create the sdist.
• Paths are specified as relative to the directory setup.py and MANIFEST.in reside in.

package_data + MANIFEST.in:

• For creating source and binary distributions completely independently of each other. Be careful.
• Files specified as package_data are included into binary distributions and installation.
• Files included by MANIFEST.in are included into the sdist.
• The setup() option include_package_data must not be set.

Implementation notes

0. This project does not assume that the end user has Cython installed; if it isn't available, it will use existing C source files, and fail if they don't exist. If Cython is available, the latter will be used to build the cython extension. This will in particular update C source files if .pyx files were updated. The implementation follows this StackOverflow discussion; also see item 2 below.

   Finally, all generated C source files are included in the resulting distribution (for both sdist and bdist).

1. In Cython projects, it is preferable to always use absolute module paths when absolute_import is in use, even if the module to be cimported is located in the same directory (as the module that is doing the cimport). This allows using the same module paths for imports and cimports.

   The reason for this recommendation is that the relative variant (from . cimport foo), although in line with PEP 328, is difficult to get to work properly with Cython's include path.

   Our setup.py adds ., the top-level directory containing setup.py, to Cython's include path, but does not add any of its subdirectories. This makes the cimports with absolute module paths work correctly[8] (also when pointing to the library being compiled), assuming mylibrary lives in a mylibrary/ subdirectory of the top-level directory that contains setup.py. See the included example.

2. Historically, it was common practice in setup.py to import Cython's replacement for distutils' build_ext, in order to make setup() recognize .pyx source files.

   Instead, we let setuptools keep its build_ext, and call cythonize() explicitly in the invocation of setup() if Cython is available on the target system. If not, we rely on setuptools' feature to compile C source files instead of .pyx files which seems to happen transparently in recent versions. It is also the approach given in Cython's documentation, albeit it refers to distutils instead of setuptools.

   This gives us some potential bonuses:

   • Cython extensions can be compiled in debug mode (for use with cygdb).
   • We get make-like dependency resolution; a .pyx source file is automatically re-cythonized, if a .pxd file it cimports, changes.
   • We get the nice [1/4] Cythonizing mylibrary/main.pyx progress messages when setup.py runs, whenever Cython detects it needs to compile .pyx sources to C.
   • Note that there is still no need to keep the generated C files under version control; the respective C source files will be generated by setup.py on the developers machine whenever needed; they will also be included in binary and source distribution packages.

   The approach used here relies on setuptools' automatic switching between Cython and C compilation depending on whether Cython is installed, although in a slightly different way than described in the Cython documentation (we still run cythonize under our own control, but switching to C sources will happen if Cython isn't available).

   This may hve the effect that, since setuptools does not see the Cython source files, it will not package them by default; hence the use of package_data in setup.py to package both the .pyx and .pxd files.

   One of the side effects is that cythonize() will run even if the command-line options given to setup.py are nonsense (or more commonly, contain a typo), and before control is passed to setup(). Thus, don't go grab your coffee until the build starts compiling the generated C sources.

   For better or worse, the chosen approach favors Cython's own mechanism for handling .pyx sources over the one provided by setuptools.

3. Using setuptools with Cython projects needs setuptools >= 18.0, to correctly support Cython in requires[9].

   In practice this is not a limitation, as 18.0 is already a very old version (38.0 being current at the time of this writing).

Distributing your package

If you choose to release your package for distribution:

0. See the distributing section of the packaging guide, and especially the subsection on uploading to PyPI.

   Especially if your package has dependencies, it is important to get at least an sdist onto PyPI to make the package easy to install (via pip install).

   • Also, keep in mind that outside managed environments such as Anaconda, pip is the preferred way for installing scientific Python packages, even though having multiple package managers on the same system could be considered harmful. Scientific packages are relatively rapidly gaining new features, thus making access to the latest release crucial.

     (Debian-based Linux distributions avoid conflicts between the two sets of managed files by making sudo pip install install to /usr/local, while the system apt-get installs to /usr. This does not, however, prevent breakage caused by overrides (loading a newer version from /usr/local), if it happens that some Python package is not fully backward-compatible. A proper solution requires one of the virtualenv tools at the user end.)

1. Be sure to use twine upload, not python -m setup upload, since the latter may transmit your password in plaintext.

   Before the first upload of a new project, use twine register. As of August 2017, pre-registration of new packages is no longer needed or supported; just proceed to upload. See new instructions.

   Official instructions for twine.

2. Generally speaking, it is a good idea to disregard old advice on Python packaging. By 2020 when Python 2.7 support ends, that probably includes this document.

   For example, keep in mind that pip has replaced ez_setup, and nowadays pip (in practice) comes with Python.

   Many Python distribution tools have been sidelined by history, or merged back into the supported ones (see the StackOverflow answer already linked above). Distutils and setuptools remain, nowadays fulfilling different roles.

Compatibility

Tested on Linux Mint, Python 2.7 and 3.4.

On Mac OS, the data_files approach used in the example will not work. See other options for packaging non-package data files above.

Not tested on Windows (please send feedback, e.g. by opening an issue).

Platform-specific notes

On Linux Mint:

• The package installs into a subdirectory of the base install location, with a name following the format setup_template_cython-0.1.4-py3.4-linux-x86_64.egg. The mylibrary and test subdirectories appear under that, as does this README.
• with python3 setup.py install --user, the base install location is $HOME/.local/lib/python3.4/site-packages/.
• with sudo python3 setup.py install, the base install location is /usr/local/lib/python3.4/dist-packages/.

Then, in Python, import mylibrary will import the library. The test subdirectory of the project is harmless; import test will still import Python's own test module.

License

The Unlicense

Includes short code snippets based on StackOverflow answers; for attribution, original links are provided in the source code.