Installing as a module from PyPI leaves installations vulnerable to bugs in dependencies

[richvdh]

One of our recommended installation methods suggests installing Synapse as a module from PyPI.

This works fine, but if one of our downstream dependencies has a bug (such as a security vulnerability) then it is difficult for administrators to know they need to update their installation. This in turn leads us to have to increase our minimum dependency versions (see #13172 for example), which makes things difficult for downstream packagers.

[richvdh]

Ideas for solving this:

• stop recommending pip as a way to install synapse, and support only docker/debs
• have poetry export a requirements.txt and get users to use it on upgrade somehow
• decide that we no longer want to support downstream packagers and just pin to a single version of all our dependencies

[DMRobertson]

stop recommending pip as a way to install synapse, and support only docker/debs

If we do this, I think people will end up pip installing as long as there is a matrix-synapse on PyPI.

[DMRobertson]

I wouldn't object to a situation where

• the Synapse source code (pyproject.toml) has broad dependencies for downstream packagers' benefit
• the packages we upload to PyPI have locked dependencies

but I don't know how easy that would be to orchestrate.

[richvdh]

stop recommending pip as a way to install synapse, and support only docker/debs

If we do this, I think people will end up pip installing as long as there is a matrix-synapse on PyPI.

Yes, but the first problem here is that we are actually recommending an approach that leaves people in a mess. It's entirely reasonable to address this in two steps (where the second step may or may not happen any time soon):

• first, stop recommending a broken approach
• second, worry about people who follow other approaches that we don't recommend.

[richvdh]

Ideas for solving this:

* stop recommending `pip` as a way to install synapse, and support only docker/debs

* have poetry export a `requirements.txt` and get users to use it on upgrade somehow

* decide that we no longer want to support downstream packagers and just pin to a single version of all our dependencies

It seems surprising to me that there is no poetry-blessed way of doing this.

[DMRobertson]

• the packages we upload to PyPI have locked dependencies

python-poetry/poetry#2778 would help here.

[richvdh]

stop recommending pip as a way to install synapse, and support only docker/debs

We'd better come up with a decent answer to #9944 if this is the route we want to take.

[dklimpel]

There is an other option.
Recommend to create for every version a new virtual env and change the link to the new virtual env. That is also an easy way to role back if there is no db change.

[reivilibre]

A further option is that we could publish one package on pip that has locked deps (to support people that want to use pip for installation) and one that doesn't (to support packagers that would find that a nightmare?).

Maybe I'm the only person using pip out of preference, but the above would at least seem to solve the situation.

[DMRobertson]

It might be instructive to look for other standaloneish applications distributed on PyPI and see what they do to handle this situation. I'm struggling to think of a good example though. (Jupyter, black, isort, mypy all sprang to mind, but they have to coexist with devtools so I imagine they have lax requirements.)

[richvdh]

I think other comparable standaloneish applications simply aren't distributed via PyPI, and people are expected to use docker images or os-specific packages. Which amounts to my first suggestion: "stop recommending pip as a way to install synapse, and support only docker/debs".

[sbiberhofer]

Just to chime in here from a distribution packager's perspective: At least on FreeBSD we basically repackage the PyPI release, so locking dependencies on the released package would probably force us to move to the source release. We can, of course, always switch over and build our packages from there, but a minor heads-up would be appreciated if you go that route as that release doesn't provide a legacy setup.py that we currently rely on for reasons completely unrelated to the issue at hand.

[DMRobertson]

as that release doesn't provide a legacy setup.py

poetry-core generates setup.py for us when we build sdists and wheels. python -m build should be all you need to generate an identical setup.py from source.

[gsemet]

We distribute a locked requirements.txt but it is not easily scalable. Need to be able to generate a -frozen version of the wheel. Should be easily done by repackaging the wheel I guess.

[gdt]

In general, I don't think it is the responsibility of synapse to save people from not having updated things that synapse depends on. Forcing dependencies for this causes a lot of trouble in packaging, and packaging is how almost all instances of widely-used software is used. I think synapse should consider packaging systems as the main target of releases. (That's avoiding specifics because I don't have them paged in, but this general issue has been a recurring problem.)

[gsemet]

We switched now to packaging our application using zipapp. This only requires a local interpreter. This is good solution to avoid dependency nightmare, and ensuring our user uses the same version of the dependencies we have carefully validated.