New release on PyPI?

[pganssle]

Since this library is vendored by pip, which is itself vendored in CPython's ensurepip module, the fact that there's no release that includes PR #403 is blocking the removal of the deprecated abstract base classes from the collections module, long advertised as happening in Python 3.8, see python/cpython#10596

Normally I am the last person to ask for releases (I assume maintainers have good reasons for not doing so, and I usually don't like being on the receiving end of such requests), but the deadline for the 3.8 beta release is coming up very soon, so if there's any way to expedite an html5lib release it would really help that effort, and it would avoid stickier solutions like patching html5lib directly in pip.

[vstinner]

This issue prevents to remove ABC aliases from the collections module in Python 3.9: https://bugs.python.org/issue37324

[justinmayer]

Now that Python 3.8 has been released, I imagine the urgency for releasing a new version to PyPI has become somewhat elevated.

As a side note, this tendency for package releases to lag behind master is one that I have personally encountered dozens of times... including when the fault is entirely my own, in projects that I maintain! It's a shame because it means there are features and bug fixes in master that nearly nobody is benefiting from, and all because publishing new releases to PyPI is just cumbersome enough to prevent maintainers from prioritizing it. (And once again, I'm including myself in that group.)

I recently solved this problem by creating AutoPub, an open-source project that helps automate package releases on PR merge, which I recently announced during my EuroPython talk a few months ago in Basel, Switzerland. Rather than explain its many benefits here, you can read more about them on the AutoPub project page.

Since the announcement last month, I've added AutoPub to several projects I maintain:

• Pelican — PR adding AutoPub
• django-elevate — PR adding AutoPub
• django-autoslug — PR adding AutoPub

... and, of course, AutoPub releases itself via AutoPub.

As a maintainer, I have been thrilled with the results. I would be happy to submit a PR to this repository that automates package releases via AutoPub and Travis, assuming that would be of interest. What do you think, @gsnedders ?

[vstinner]

https://bugs.python.org/issue37324 has been fixed: collections ABC aliases have been removed from Python 3.9. As a consequence, html5lib is no longer compatible with Python 3.9.

How can we help you to get a new release?

[vstinner]

The issue broke pyperformance benchmark suite which has a benchmark on html5lib:
https://bugs.python.org/issue38739

[hugovk]

I've reviewed some PRs and left some comments on others, and made #430 which should help make releasing easier.

(Although I've just realised this is similar to Justin's suggestion so don't mean to step on any toes! #419 (comment))

Also, perhaps a transfer to JazzBand or the PSF may help unblock some of the more basic maintenance tasks such as releasing?

• https://jazzband.co/
• https://github.com/psf

[ghost]

The collections issue outlined by @vstinner currently is causing tools such as poetry for dependency management to not be able to be used on Python3.9 as per python-poetry/poetry#1733, which will block many projects from being able to maintain support for newer python versions unless an alternative to HTML5LIB can be found currently, which would be most unfortunate...

[vstinner]

I contacted @jgraham and @gsnedders by email to see how I can help them to publish a new release.

[dimaqq]

@vstinner did you hear back from @jgraham or @gsnedders perhaps?

Otherwise, if all what's needed is to fork a repo and publish under a different pypi name, then that should not be a big problem :)

[vstinner]

@vstinner did you hear back from @jgraham or @gsnedders perhaps?

No.

Otherwise, if all what's needed is to fork a repo and publish under a different pypi name, then that should not be a big problem :)

There is a process to request to get the ownership of an existing PyPI name: see https://www.python.org/dev/peps/pep-0541/#abstract

[frenzymadness]

html5lib is also bundled in bleach

[hugovk]

@jgraham @gsnedders Thank you both very much for your work on this project!

It's been two years since a release and this project is widely used (2.5m downloads last month according to https://pypistats.org/packages/html5lib!), and it'd be great to keep things ticking over.

I understand if you've moved on and have other interests. Please would you consider giving merge and PyPI rights to one or more extra maintainers?

I would like to volunteer to help get a new release out, to unblock the collections.abc issue, and for general maintenance tasks, at least as an interim maintainer like @willkg was for the 1.0 release, and would welcome other volunteers too.

Otherwise, I suggest we make a community fork (I've made a start at html5lib-python/html5lib-python) and apply for a PEP 0541 transfer.

Again, thanks for all your work on this project!

[carltongibson]

Hi @hugovk.

I think at this point it's best to push forward with your fork. I see you've pulled in some the PRs here that are ready. It may be worth commenting on each that that's the case, and perhaps a ticket on the fork to track that progress?

I think beginning the transfer request would be wise. (I don't suppose those are instant.)

Thank you for your effort.

[hugovk]

PEP 0541 requires PyPI maintainers attempt contact at least three times over six weeks: https://www.python.org/dev/peps/pep-0541/#reachability, so I've opened a request at pypi/support#227. Although it would be preferable to continue things here.

I've commented the PRs here that have been merged into the fork.

Thanks!

[carltongibson]

Hi @hugovk. Super stuff. Thank you again.

Maybe it's worth opening a PR from your master branch to back here, so that at least there's the possibility that your effort can be picked up, if a sustainable solution here does turn up. 🤷‍♂️

[gsnedders]

Hi. I've basically been away from all tech stuff for the past month, and not seen any of the recent discussion.

To summarize, somehow I missed the offer of @jdufresne offering "to help out on a very limited time basis" in the email thread last October and was already on holiday by the time of @vstinner's email in December. Sorry!

Some of my disincentive since last summer has been undergoing a big move in my life, and spending very little time unpaid on tech stuff (as mentioned before, I'm more than happy to spend more time on html5lib if I'm paid to do so; I have relatively little interest in maintaining a major project used by enough for-profit companies without any compensation). That said, let's try and find a workable path forward.

What I said in October in email thread a bunch of you were on then still holds true:

I think my suggestion for a short-term path forward is we get some agreement on adding one or two maintainers (and preferably not just for the next release, because then we're back here in a year's time), we get master passing CI (inc on 3.8 and CPython master), and make a release. I think with other people involved I'm more likely to spend time on html5lib (me eventually landing changes after not getting anyone to review them for months, which has happened before, is a bit demotivating!), so it's probably worthwhile to assume that I'll be a part of whatever comes next.

So, let's:

• Figure out what PRs need to land prior to a next release (and I'll try and get more done on train later too).
• Document how to make a release (this has been another disincentive against me making a release, not knowing what I'm doing!).
• Make a release!
• Document a bunch of policies (there's enough users and institutional knowledge between myself and @jgraham that we should write a bunch of this down); Document Python version support policy #439 is an obvious one where we are at least constrained by being vendored by pip (and I think both us and those working on pip would rather avoid them needing to fork the last supported release!).
• At least add a few committers here (and hopefully with documentation as to how to I'll be much better at/willing to make releases!) and see how that goes, and probably later add some/all as maintainers on PyPI.

Though I will say I still feel like dropping the collections.* aliases is a complete waste of time, it saves maintaining 15 lines in CPython and causes how many hours in way of work and changes throughout the ecosystem? Sure, it may be fractionally cleaner, but it's a breaking change that therefore necessitates work from plenty of Python developers rather than the Python developers supporting the ecosystem. I'd be much more understanding if it made a significant impact on maintenance work on CPython, but I can't imagine those 15 lines do. Is the benefit of breaking existing code really justified in this case?

Let me know if this seems workable, either later this evening (and I realise many of you are in Europe and have previously been commenting during business hours) or over the next few days. In the mean time, I'll try and land a fair number of the open maintenance-task PRs.

[sethmlarson]

I'm more than happy to spend more time on html5lib if I'm paid to do so; I have relatively little interest in maintaining a major project used by enough for-profit companies without any compensation).

FYI: Tidelift is willing to pay ~$2,700 / year for a maintainer of html5lib :)

[ghost]

Though I will say I still feel like dropping the collections.* aliases is a complete waste of time

Why would it be a waste of time to drop features that are no longer supported? Or do I misunderstand? Should we not be encouraging devs to use the most recent versions of python, rather than old versions?

[gsnedders]

Why would it be a waste of time to drop features that are no longer supported? Or do I misunderstand? Should we not be encouraging devs to use the most recent versions of python, rather than old versions?

Backwards incompatible changes have a cost on the ecosystem as a whole. The question is whether the cost of maintaining those 15 lines of code in CPython is higher than every Python project that uses the deprecated updating to the (new) alternative and then making releases, and therefore providing a blocker to end-users from upgrading their installation of CPython if their scripts don't support it. Every backwards-incompatible change has a cost on the ecosystem, and encourages users to use older versions of Python that work with their scripts rather than more recent versions.

[gsnedders]

FYI: Tidelift is willing to pay ~$2,700 / year for a maintainer of html5lib :)

And that is… not much in terms of contractor rates, such as those I charge most of my clients.

Some of my hesitancy in "lifting" that is both in terms of who will do code review (esp. if it's just me and not @jgraham around here), and lack of compensation for anyone doing code review (because really it should be split across anyone not otherwise paid to work on the project!), along with needing to get around to speaking with accountant about tax implications of doing so.

[vstinner]

Though I will say I still feel like dropping the collections.* aliases is a complete waste of time (...)

FYI aliases to ABC in collections have been reverted in Python 3.9, but they will be removed again in Python 3.10: https://bugs.python.org/issue39674

But it doesn't solve the lack of new html5lib release ;-)

[vstinner]

Figure out what PRs need to land prior to a next release (and I'll try and get more done on train later too).

The initial issue was collections ABC, but this issue has already been fixed in html5lib master: 4f92357

[hugovk]

@gsnedders @jgraham Hi, I created https://test.pypi.org/project/html5lib/ earlier for deploy testing, if you create accounts on TestPyPI, I'll give you the full rights.

[gsnedders]

OK, so status update from what I've done over the past few weeks:

• We now have support for pytest 4 and 5, which gets rid of one of the most common distro patches.
• We should land Don't use cElementTree on Python 3 #502 which avoids an import-time DeprecationWarning in 3.10.
• We should land Deprecate the sanitizer and recommend Bleach #501 which documents the sanitiser as being deprecated.
• And also excitingly, Add benchmarks and optimize html5lib a bit #493 gives pretty significant speed gains when parsing small documents and minor ones which larger documents.

Hopefully we'll get those three landed this week and make a release by the end of the week.

If I have the chance, I might try and fix #96 for wide Python builds only.

[EmilStenstrom]

@gsnedders You are awesome! :)

[FrankDMartinez]

@gsnedders The end of the week has come; any word?

[gsnedders]

We've now landed all of the above, I think the changelog has all the potential breakage/notable changes, so… let me try to make a release tomorrow?

[gsnedders]

OK, that's 1.1 shipped. Shout at me if anything is broken (in a new issue, please!).

(I realise it's now Tuesday, so I failed at "tomorrow" as well as the end of the week!)