author: Stuart Mumford, Thomas Robitaille
date-created: 2019 March 7
date-last-revised: 2019 March 7
type: Process
status: Discussion
The astropy-helpers package contains a collection of utilities intended to help build and distribute the core astropy package, as well as coordinated and affiliated packages in the Astropy ecosystem, and it is also used by packages outside of the ecosystem (at the time of writing, almost 400 repositories on GitHub appear to use astropy-helpers).
The original APE proposing the creation of astropy-helpers (APE 4) was accepted in June 2014. Since then, the state of the Python packaging ecosystem has changed significantly, with advances such as binary wheels that can be provided for all major platforms, tools such as tox and setuptools_scm, and more recently support for PEP 517 and PEP 518 (in pip 19.0 or later), and many incremental updates and new features to setuptools (in setuptools 30.3 or later). The purpose of this APE is to show how by relying on these tools, we can significantly simplify the packaging for astropy and all the affiliated packages and other packages using astropy-helpers, and remove the need for astropy-helpers altogether in pure-Python packages.
We now go through the various aspects of package infrastructure that astropy-helpers has solved in the past and how we can now address these instead using standard packaging tools.
We first take a look at what changes are needed for all packages, regardless of whether they have compiled extensions or not. For pure-Python packages, if all these changes are made, astropy-helpers is no longer needed and can be removed from the repository.
Astropy-helpers provides the ability for anyone to run tests in the source distribution to do this with the:
python setup.py test
command, which behind the scenes built the package, installed it into a temporary directory, installed pytest-astropy if needed, and ran the tests there.
However, this can now be done more cleanly using tox. Tox is a tool that
makes is easy to install packages into a temporary clean Python environment
along with the dependencies declared in setup.cfg, and run a set of custom
commands - in a sense it does what the test command from astropy-helpers did,
but in a more general way (including the use of an isolated test environment).
For example, by putting the following into tox.ini:
[testenv:test] deps = pytest-astropy commands = pytest aplpy
developers will then easily be able to run the tests with:
tox -e test
The benefit of using tox over just running pytest directly is that since behind the scenes it creates a source distribution of the package and installs it into a clean environment, it ensures that source distributions for the package are fully functional and that all required files and dependencies are declared. For packages with C/Cython extensions it should also guarantee that they are always rebuilt before running the tests.
However, for pure-Python packages, developers can also opt to simply run the tests directly with pytest, e.g.:
pytest packagename
so using tox is optional.
We note that these changes have no impact on the availability of the
package.test() function which is unrelated to astropy-helpers and relies
instead on the astropy core package to provide a test runner.
Astropy-helpers provides a python setup.py build_docs command that
behind-the-scenes built the package then added it to sys.path, then ran the
documentation build. Having the package be importable is needed as documentation
often includes API sections that are dynamically created based on the package.
As for testing, we can accomplish the same process more cleanly instead using tox - a minimal tox configuration might look like:
[testenv:build_docs] deps = sphinx-astropy commands = sphinx-build docs docs/_build/html -W -b html
and this would be run using:
tox -e build_docs
Developers not wishing to use tox could also accomplish the same by doing:
pip install -e . cd docs make html
These changes will have no impact on ReadTheDocs as that service never made use
of the build_docs command, instead invoking sphinx directly with
sphinx-build.
Currently, astropy-helpers includes version helpers that take the version
defined in setup.cfg or setup.py and add a developer string when in a
checked out version of a git repository. The developer string consists of
version.devN where N is the number of commits since a release.
The Python Packaging Authority (PyPA) now provide a setuptools extension called
setuptools_scm which can entirely replace the version helpers in
astropy-helpers. The way this package works is that versions are no longer
specified in setup.cfg or setup.py - instead the versions are taken from
tags. Developer version strings produced in this way are much more sophisticated
and can indicate for example if the working copy is clean or has local changes,
and whether it is a stable tagged version or a developer version. We note that
the default way of setting up setuptools_scm results in the version string not
being updated automatically in ‘editable’ installs of a package (i.e. pip
install -e .), however a workaround is available.
Switching to setuptools_scm requires minimal configuration, which is well
described in its documentation and we therefore do not repeat here. However, one
subtlety is that since it relies on tags to determine versions, for packages
such as the core astropy package which have all their tags on branches (at least
in recent years), we will need to add a ‘developer’ tag on master straight
after branching, e.g. after creating a v4.0.x branch we should tag the next
commit on master as v4.1.dev.
At the moment, package data and entry points for packages can be defined via
get_package_data and get_entry_points in setup_package.py files.
However, this adds unnecessary complexity, as even for the core package it is
simple to define the data and entry points in setup.cfg using the
[options.entry_points] and [options.package_data] sections, e.g.:
[options.entry_points]
console_scripts =
fits2bitmap = astropy.visualization.scripts.fits2bitmap:main
...
[options.package_data]
astropy = astropy.cfg, CITATION, **/data/**/*
...
Removing the ability to specify package data in setup_package.py files removes the dependence of the setup.py sdist command on astropy-helpers, which is essential to using the PEP 518 build time dependencies discussed below.
Using setup.cfg to define package data and other options does rely on
setuptools 30.3 or later, which is now over two years old. Nevertheless, to
minimize issues for users with older Python installations, we recommend
including a version check for setuptools inside the setup.py file.
A little-known feature of astropy-helpers is the ability to define hooks in
setup_package.py files for steps to be carried out before/after specific
setup.py commands. We propose removing this functionality since this feature
was never well advertised and likely only used in the core astropy package, and
our proof-of-concept implementation in the core package shows that we can easily
remove this.
One of the common issues that packages have regularly run into and which PEP
518 solves is how to define dependencies required for building a package. The
setup() function provided by setuptools takes a setup_requires argument
which can include dependencies for the build, but unfortunately the setup.py
file has to be executed before the setup() function is run, which leads to
the circular issue that any code in setup.py can’t rely on packages
specified in setup_requires - thus, setup_requires does not properly
solve the issue of build-time dependencies since those dependencies are only
installed part way through the build process.
PEP 518 instead specifies that build-time dependencies can be specified in a
file called pyproject.toml that contains for example:
[build-system] requires = ["setuptools", "wheel", "numpy==1.13.3"]
Provided that the package is installed with a tool such as `pip`_ which understands this file, the build-time dependencies will be installed before the setup.py file is executed. PEP 517 takes this concept further by specifying that the build should happen in an isolated environment, which means that one could specify a pinned version of cython or jinja2 to use even if a different version is installed in the user’s environment.
With this in mind, if astropy-helpers was still needed, it could therefore now
be included as a build-time dependency in pyproject.toml which removes the
need for the ah_bootstrap.py file and the git submodule. Furthermore,
astropy-helpers could be pinned to specific versions in pyproject.toml and
different packages could use different versions (thanks to the build isolation,
this will not be a problem).
For packages that need Numpy to be built, numpy should also be included in
the list of build-time dependencies in pyproject.toml. Currently, when
defining C extensions that need to use numpy, we need to add
numpy.get_include() to the include_dirs argument of Extension.
However, this can’t be done until numpy is installed, so astropy-helpers
currently provides a way for packages to specify the string ‘numpy’ instead,
and replacing it with numpy.get_include() on-the-fly. This workaround will
no longer be needed once Numpy is specified in pyproject.toml since Numpy
will be installed before the extensions are defined, and we suggest that
packages should instead specify numpy.get_include() explicitly.
Note that Numpy should be pinned to the oldest compatible version in the
pyproject.toml file - this is because if a user does pip install astropy
numpy==1.14.2, the pinning of Numpy only applies to the version installed
after astropy has been built, and the version taken to build astropy is taken
from the pyproject.toml file. This means that packages such as astropy have
to be built with the oldest compatible version of Numpy since the build will
then be forward-compatible with any later version of Numpy (this is similar to
the approach taken for conda packages). In addition, the oldest version should
be that for which wheels are available so for packages where this depends on
Python version, environment markers can be used (see PEP 508), e.g.:
"numpy==1.13.1; python_version<'3.7'", "numpy==1.14.5; python_version>='3.7'",
A side benefit of this is that with these pinnings in place, building wheels
with pip wheel . will automatically create wheels compatible with all
available versions of Numpy.
Note that setup_requires should no longer be used in
setup.py/setup.cfg, and install_requires should require numpy to be
>= than the oldest version mentioned in pyproject.toml.
Currently, astropy-helpers includes functionality to auto-generate C code from
Cython extensions and ensure that when packages are released, the C code is the
one used to compile extensions, even if Cython is installed. This was originally
done to make sure that Cython was not required to install astropy, and to avoid
issues due to differences in the generated C code from different Cython
versions. When releasing stable versions of the core package for example,
developers had to remember to run the build command before sdist to
include the generated C code, otherwise this would cause issues for users that
didn’t have Cython.
However, Cython should now be included as a build-time dependency in
pyproject.toml and developers should not include generated C code in
released packages. Build-time dependencies in pyproject.toml files are
always installed from wheels, so this would not have a significant performance
impact for source distributions - and since most users installing astropy with
pip will be installing astropy wheels, this will have no impact for most users.
Note that if needed, we can even pin the Cython version in pyproject.toml to
ensure consistency across all builds. With this in place, the custom
build_ext command in astropy-helpers can be removed.
Astropy-helpers provides a way for developers to use setup_package.py files
throughout a package to define extensions and definitions for external
libraries. This is one of the only parts of astropy-helpers which we think it
makes sense to preserve, and we argue that it is so general that it should be
released as a package with a more generic name than astropy-helpers, such as
extension-helpers - this will allow us to also avoid breaking astropy-helpers
and instead starting fresh with a clean package (although the git history could
be preserved).
However, we note that for small packages, developers can also simply define
extensions inside setup.py, which would mean that astropy-helpers (or
extension-helpers) would not needed for these packages either.
- The pull request astrofrog/astropy#83 shows the changes necessary for the astropy core package, and include an experimental version of extension-helpers.
- This SunPy branch shows how that package can use astropy_helpers only as a build time dependency for building a C extension
- The following pure-Python packages have dropped astropy-helpers and adopted
some or all of the recommendations outlined here:
- DKIST (3552bbeb)
- MOSViz (spacetelescope/mosviz#179)
- CubeViz (spacetelescope/cubeviz#492).
- aas-timeseries (aperiosoftware/aas-timeseries#492)
All the changes to packages are already described above, but to summarize, the following table shows the correspondence between old astropy-helpers features and their proposed replacement:
| Feature | Replacement |
astropy.version_helpers |
setuptools_scm |
Package data in specification
in setup_package.py files |
All package data should be
specified in setup.cfg |
python setup.py test |
tox or direct use of pytest |
python setup.py build_docs |
tox or make html
or sphinx-build |
| Delayed import of Numpy when building C extensions | Replaced by build dependencies and isolation in PEP 517 and PEP 518 |
| Transpilation of Cython to C before sdist | |
| Package specific versions of astropy-helpers provided by git submodule |
We propose that a new package called extension-helpers be created starting from
astropy-helpers but with only the minimal amount of functionality needed to
handle the definition of compiled extensions, external libraries, and the
auto-discovery of Cython extensions. This package would need to be declared as a
build-time dependency in pyproject.toml.
In addition to the changes described here, we also recommend moving all or as
many as possible of the options for the setup() function in setup.py to
the setup.cfg file as described in the setuptools documentation.
Many of the changes described here can already be safely made now. The only
change that requires some consideration in terms of timeline is the use of
pyproject.toml (which is only needed for package with compiled extensions).
Full support for this file, including the environment markers (which allows
different Numpy dependencies for different Python versions for example) and the
build isolation, only became available in pip 19.0 onwards.
Therefore, we recommend that Pure-python packages can make all the changes
described here now. On the other hand, packages that need to rely on
pyproject.toml for building C extensions should make the switch at the time
they are happy to rely on pip 19.0 (released 2019-01-22) for installs from
source distributions, but they can still make other changes now, e.g. using
setuptools_scm or making a greater use of setup.cfg.
We note however that provided that wheels are available for a package with
compiled extensions, it may be acceptable to transition to using
pyproject.toml sooner rather than later because pip 19.0 would only be
required for source installs, but since most users that pip install would be
using wheels, this may not be an issue.
If done properly, these changes should have no noticeable impact for users.
Users will still be able to pip install (or conda install when available)
packages and for large packages such as the core astropy package, which contains
a lot of compiled extensions, most users will not see any
difference since they will be installing astropy from pre-built packages (wheels
or conda packages). Running tests using package.test() will still be
supported.
The immediate impact for developers of packages that use astropy-helpers is having to update the layout and infrastructure in their packages to follow the new guidelines presented here. However, we emphasise that these changes are recommendations and not mandatory, and astropy-helpers will continue to work as expected as long as it is included as a submodule. However, we recommend that astropy-helpers no longer be actively developed, in which case guarantees could not be made that some aspects of the astropy-helpers infrastructure will not break with future releases of Python, setuptools, or sphinx for example.
However, we believe that the initial effort to switch over to the new guidelines
will be a worthwhile investment - in all cases it will mean being able to get
rid of astropy-helpers as a submodule and the confusion and headaches this can
cause, and it will make for example the definition of package data much simpler
and not have to worry about setup_package.py files in most cases.
The astropy package-template will be updated to reflect the latest recommendations, which will make it easier for developers to update their packages.
Users who wish to contribute fixes to the core astropy package or other packages
will be encouraged to have tox installed if they want to easily run
tests or build documentation locally. However, this is an easy package to
install with `pip`_ and we could also add code in setup.py so that running
e.g. python setup.py test or python setup.py build_docs gives a helpful
error message with instructions on updating and using tox to run tests and build
the documentation.
We note however that using tox is just a convenience and will not be compulsory, especially for pure-Python packages where running pytest directly will work.
One of the main benefits of these changes will be to not have to maintain astropy-helpers any longer. Some parts of astropy-helpers have relied on hacks that can be brittle and break with new setuptools or Sphinx releases. Maintenance will still be needed for the proposed extension-helpers package but this will be a much smaller package than astropy-helpers, and by making it more generic and usable by any package, we hope to attract contributions from beyond the Astropy team.
By relying on standard packaging infrastructure, this should facilitate the job of people involved in package managers - for example, the conda-forge infrastructure currently recommends that recipes should use pip to build packages, but this is not possible or easy at the moment for packages using astropy-helpers since they need to pass custom flags to setup.py to prevent the default behavior of checking for astropy-helpers releases online.
The changes described here are opt-in, so barring any breaking changes in Python, setuptools, or Sphinx, everything should continue to work as expected if developers do not make any changes.
In the words of Stuart: Do nothing, suffer submodules and maintaining the spaghetti code of astropy-helpers until we demoralise the whole community and Julia takes over.
<To be filled in by the coordinating committee when the APE is accepted or rejected>