ci, packaging, meta: modernize Python packaging for Sopel

[Exirel]

Description

After hours of reading various documentations (mostly: the packaging user guide and setuptools guide for pyproject.toml), I think I end up with a pretty good understanding of the topic matter to modernize Sopel's build system. I also read various PEP, such as PEP 517, PEP 518, PEP 643, PEP 660, with various results.

How to review this PR

You have various options: I advise to, at least, try out a pip install -e . locally to see if it works for you.
The best thing would be to read all the links and do some research to have all the knowledge needed, but since it might take a lot of your time, I tried to write down all the important bit in this PR's description.

Take your time to read it fully, and ask me questions on IRC.

Now and then

At the moment, Sopel's build system is composed of the following steps:

• local install is done through either pip install -e . or python setup.py develop, then requires pip install -U -r dev-requirements.txt
• build the source and built distribution with python setup.py sdist bdist, generating a .tar.gz and a .whl
• upload with twine
• user install is done through pip install sopel

The goal of this PR is to switch to the more modern and standard approach to build package in Python:

• local install is done through pip install -e ., then requires pip install -U -r dev-requirements.txt
• build the source and built distribution with python -m build, generating a .tar.gz and a .whl
• upload with twine
• user install is done through pip install sopel

The workflow doesn't change (dev, build, upload, install), and the tools used for development and build are more standardized. Curious reader can have a look at the official tutorial for Python packaging, which contains the same workflow and tools.

Switching to pyproject.toml

Technically, we don't need to switch from setup.py to pyproject.toml in order to use modern tools, such as build. However, the modern packaging is now standardized around that file, and there is no way around it: setup.py is an outdated tools that served us well for almost two decades (distutils was created around 2000 and setuptools 2004) and need to be retired.

The pyproject.toml contains two important sections:

• [build-system], following PEP 517 and 518, to declare which build backend must be used to generate source and built distribution
• [project], following PEP 621 and official documentation, to declare most metadata from setup.cfg/py into a normalized format

I decided to keep using setuptools as our backend of choice: this is the default backend used by build, ensuring that we don't switch to a complete new set of tools and configuration files. I ensured that everything was also available in pyproject.toml, with minor adjustments:

• finding the appropriate package with setuptools through [tool.setuptools.packages.find] (backend specific)
• remaining metadata still present through [tool.setuptools]
• requirements.txt is replaced by the now standard project.dependencies option
• entrypoints (both console and custom) are properly defined
• [project.urls] now contains more than the homepage

Removing setup.py, keeping setup.cfg

As a result of using pyproject.toml with the setuptools backend, I was able to completely remove the setup.py file. However, since we are still using setuptools, I was forced to keep the setup.cfg file. Setuptools's documentation points out that one may be forced to keep an empty setup.py file, or that setuptools will generates it if you have a setup.cfg file. Confusing? Yes. I know, setuptools is improving by the minute, so it's a small price to pay to be on track with the future of Python packaging.

I don't think it is relevant to perform various Python check ourselves: let the installer of the user do that for us. The Python requirement is now standard, so one should not be able to install Sopel with an outdated version of Python.

The setup.cfg file that's left is now empty of project's metadata, and contains only configurations for our development tools, such as flake8 or mypy.

As a result, a developer will need to issue the following command to work on Sopel locally: pip install -e . && pip install --upgrade -r dev-requirements.txt.

Installing the project from source still works by following Python standard: pip install ..

About dependencies

There is a point that I'm not completely happy with: the requirements. I feel like tooling around them is not really at the top. I guess that's why PyPA suggest to use other tools than pip to develop projects, such as Hatch or Poetry. My goal with this modernization of our build system is to allow developers to use these tools if they want to, as long as we keep building it the same way for everyone.

In any case, I made two choices:

• remove the requirements.txt file, and put the requirements into the pyproject.toml file
• keep the dev-requirements.txt file, and not have a "dev requirements" into the pyproject.toml file

I need to be able to do pip install --upgrade -r dev-requirements.txt to get the up to date version of flake8, pytest, and so on, because doing pip install --upgrade -e .[dev] didn't provide that: it would keep the currently installed version as long as they conformed to the spec. Other tools than pip can do that, but it didn't seem like a good idea to change that process for now.

Updating CI configuration

This is probably the most controversial change of this PR: I decided to update the CI to reflect modern standards. First thing first, I switch to python -m build, as recommended.

Then, I upgraded the github actions to the latest version of each.

And last but not least, I decided to use PyPA's action to publish on PyPI.

As far as I know, everything works like a charm.

Removing old files

Both sopel.py and pytest_run.py are outdated tools. I don't understand why we kept them for so long, and what purpose they serve exactly. In any case, I removed them.

I updated the commit-hooks with something that should work. And if people wants to run Sopel from source, then they need to understand how they are supposed to do that: since it is now all standard and well documented, there is no reason to maintain anything more than the standard.

Improving CONTRIBUTING.md

I rewrite a part of CONTRIBUTING.md to explain with more details the typical development workflow for anyone wanted to provide a PR to the project: how to install, how to manage git, and how to run tests.

Optional: authors & maintainers metadata

I don't know what to do with these fields. Maybe for another PR?

Checklist

• I have read CONTRIBUTING.md
• I can and do license this contribution under the EFLv2
• No issues are reported by make qa (runs make quality and make test)
• I have tested the functionality of the things this change touches

[dgw]

Not possible to call out in line notes: The commit message for c686d51 doesn't make sense. Shouldn't that be something like ci: add wheel to initialization?

[Exirel]

All done. I've squashed the weird commit and pushed additional commits (without a rebase on master this time).

Should list more than just me. Many people were responsible for creating the code that eventually became Sopel, then for maintaining Sopel before I took over as maintainer, and now for helping me maintain it (including you).

That's a tricky question actually. The field "authors" isn't well specified, and no one seems to agree on what it actually means. Is it a list of active authors? A list of historical authors? A list of all the contributors? And how does "maintainers" fit into that?

What I suggest for now is to change that in a future PR, as this can be changed before the release. Right now, I don't have a meaningful opinion about that field.

[dgw]

Phone review today, but still caught some stuff. 💪

You're right about leaving the author etc. metadata for the future. See comment added to existing thread about that in TOML file.

[Exirel]

I'll let you open the issue for the list of authors/maintainers. Other than that, everything else have been applied.

I'm ready to squash.

[Exirel]

All good now. I tried to find a proper way to talk about venv and virtualenvs, and then I noticed that I should put a line about project manager tools that would create venv for developers.

Anyway, it should be good for a last round of review, and a squash. Then, I can start building Python 3.10 in the CI with that!

[dgw]

I think there's just one loose end to tie off, and a tiny grammar thing.

You can squash at the same time. GH will let me see the force-push diff, which will be enough unless you also need to rebase for some reason.

[Exirel]

Grammar: fixed.
Comment: replied.
Commits: squashed.

[dgw]

I don't see any files unexpectedly marked as unviewed, so let's call it done. Probably want to test this one as a last sanity check before merging, when I'm at a computer later… 😅

[dgw]

Right, I went through the python3 -m build process and uploaded the results to testpypi, to see how the new metadata works. Few weird things that are easy to fix. I will push a few commits that fix these, momentarily, but here's the written explanation of what I changed and why.

Full text of the license file is displayed in the sidebar.

Before:

After:

This one can be fixed by just using the "text" option to specify "Eiffel Forum License v2.0", similar to how the 7.x releases have. Put a pin in PEP 639 (deprecates/replaces License and the license-related classifiers) for later.

I tried directly using the license = "EFLv2" format, but it's not supported yet. So we'll have to take another modernization pass at the metadata once the deprecations and their replacements actually take effect I guess. (Attempt to future-proof rejected!) This is also why I haven't removed the license classifiers.

Funding links are separated

Old:

New:

This is because Warehouse/PyPI alphabetizes the project links. I would suggest using "Donate" or "Sponsor" for both GH and OC so they appear together in the list. Using "Donate" lets "Source" appear last, which I think is nice.

---

I also changed "Release notes" to point to https://sopel.chat/changelog/ (can't be seen in a screenshot, lol).

---

Also note, I often saw a warning in the output of build operations:

/tmp/build-env-s4nd2h1_/lib/python3.9/site-packages/setuptools/config/pyprojecttoml.py:108: _BetaConfiguration:
Support for `[tool.setuptools]` in `pyproject.toml` is still *beta*.
  warnings.warn(msg, _BetaConfiguration)

It's probably fine? But worth mentioning here, so future-us can see that it was noticed while testing this patch. Yes, the warning is still there on setuptools' main branch.

[dgw]

Merging this broke unstable docs—whoops!—because the build there was still reliant on setup.py. The fix was mostly on the Netlify configuration side, out of tree. f2712b1 contains the only repo tweak needed to make it work again.