Using --enable-system-site-packages is broken

[tornaria]

Using --enable-system-site-packages is broken:

• first, the version of sage_conf inside the venv is installed in such a crooked way that it will not override the one installed in the system site-packages, in spite the sage venv site-packages being listed before the system site-packages in sys.path.
• the version of sage_conf.py I have in my system doesn't define GAP_ROOT_PATHS (not a thing in 10.2). This results in an incredibly scary-looking error (to simulate try GAP_ROOT_PATHS=/ sage -c 'gap(1)' or worse GAP_ROOT_PATHS=/ sage -c 'libgap(1)').
• I don't know how to fix the search order for sage_conf so I just added GAP_ROOT_PATHS to the system sage_conf (I can't remove it since I don't want to break sage for other users)
• With that fixed, almost all tests pass except some in src/sage_docbuild and the top comment in src/sage/env.py

Originally posted by @tornaria in #36181 (comment)

EDIT: some relevant notes that are mentioned in the original post:

• I'm testing with PR Python 3.12.x #36181 at 29130ea, since my system python is 3.12
• I used ./configure --enable-system-site-packages --disable-doc

[tornaria]

More details:

Failures in src/sage_docbuild/*

These are all due to missing sphinx. Indeed, it was not installed in my system (nor was it installed in the venv, maybe because of the --disable-doc). I guess this can be fixed by adding # sage.doctest: needs sphinx at the top of the files as follows:

diff --git a/src/sage_docbuild/__main__.py b/src/sage_docbuild/__main__.py
index 0d15808a69c..acfcb8392a0 100644
--- a/src/sage_docbuild/__main__.py
+++ b/src/sage_docbuild/__main__.py
@@ -1,3 +1,4 @@
+# sage.doctest: needs sphinx
 r"""
 Sage docbuild main
 
diff --git a/src/sage_docbuild/builders.py b/src/sage_docbuild/builders.py
index a43fdda7350..0be43be307f 100644
--- a/src/sage_docbuild/builders.py
+++ b/src/sage_docbuild/builders.py
@@ -1,3 +1,4 @@
+# sage.doctest: needs sphinx
 """
 Documentation builders
 
diff --git a/src/sage_docbuild/sphinxbuild.py b/src/sage_docbuild/sphinxbuild.py
index 5ae1d2e6b10..6d5823934b8 100644
--- a/src/sage_docbuild/sphinxbuild.py
+++ b/src/sage_docbuild/sphinxbuild.py
@@ -1,4 +1,4 @@
-# -*- coding: utf-8 -*-
+# sage.doctest: needs sphinx
 r"""
 Sphinx build script
 

Failure in src/sage/env.py

This is because of a weird mismatch between ./venv/bin/python and ./sage -python:

$ ./sage -python -c 'from sage.env import SAGE_ROOT, SAGE_LOCAL;print(SAGE_ROOT);print(SAGE_LOCAL)'
/home/tornaria/src/sage/sage-git/develop
/home/tornaria/src/sage/sage-git/develop/local

$ ./venv/bin/python  -c 'from sage.env import SAGE_ROOT, SAGE_LOCAL;print(SAGE_ROOT);print(SAGE_LOCAL)'
None
/home/tornaria/src/sage/sage-git/develop/venv

[tornaria]

FWIW, #36997 (comment) goes to confirm that everything is ok with --disable-editable.

Even the sphinx failures are gone, because for non-editable builds the files in src/sage_docbuild are not tested, it seems.

[mkoeppe]

Using --enable-system-site-packages is broken

Yes, but we generally use the euphemism "experimental" for that.

It's broken on almost every tested platform -- see section "standard-sitepackages" in https://github.com/sagemath/sage/actions/workflows/ci-linux.yml
and there appears to be no effort to work on this. @orlitzky @dimpase

IMNSHO, it should not be advertised to end users at all. We already know that it does not work because we are testing it! Sending users into the frustration of trying it out and failing (obviously!) is a terrible practice, and in fact a severe regression back to the pre-2020 practice of trying to do platform support without using platform testing infrastructure.

[dimpase]

this option works for me, and for @orlitzky - I presume.

we advertise more dangerous things to users, e.g. packages which produce wrong maths results

[dimpase]

this option works on a 'generic' platform, with most packages coming straight from PyPI.

Of course it's hopeless - with our ultra-fine degree of package granularity - to accomplish this for a typical Ubuntu or Fedora. The version discrepancy is too much.

For this to work reliably one should get rid of most of this "inner" Jupyter and Sphinx dependencies.

If you must vendor jupyterlab and sphinx, vendor them as whole things, not as gazillion of pieces.

[tornaria]

Using --enable-system-site-packages is broken

Yes, but we generally use the euphemism "experimental" for that.

Which is very different from "not planned". I'm not asking for a refund, I report in good faith and then spent some time and effort in fixing issues that show up.

Is --enable-editable similarly experimental? (I gather not, since it's the default).

The worst part of this breakage is because of a bug in the editable installation of sage-conf which messes up the package search order.

This bug is really with --enable-editable, but it only manifests in a venv with --system-site-packages. Seems worth fixing not only for sage-the-distro-with-system-site-packages sake.

I insist on the following: even if each one needs or wants a different workflow, the underlying infrastructure has similar requirements (modularity, robustness, sane defaults, working out of the box with diverse systems, etc). The issue you have today may be my issue tomorrow, and vice versa.

[mkoeppe]

this option works on a 'generic' platform,

There's no such thing, as you know.

Of course it's hopeless

@dimpase Well OK, then let's please stop advertising it to users as if we'd expect it to work!

[mkoeppe]

Is --enable-editable similarly experimental?

No.

The worst part of this breakage is because of a bug in the editable installation of sage-conf which messes up the package search order.

OK, then let's focus on this one. Separate ticket and let's create a reproducer so we can report this issue to the editables project -- #36181 (comment)

[dimpase]

this option works on a 'generic' platform,

There's no such thing, as you know.

Of course it's hopeless

@dimpase Well OK, then let's please stop advertising it to users as if we'd expect it to work!

well, adding a suitable warning - fine, but hiding it - no.

It works out of the box on Gentoo linux. Why do you want to hide this?

[dimpase]

this option works on a 'generic' platform,

There's no such thing, as you know.

Why? The 'generic' platform is a venv with all needed up to date PyPI packages. No magic, it often just works.

[mkoeppe]

@dimpase Well OK, then let's please stop advertising it to users as if we'd expect it to work!

well, adding a suitable warning - fine, but hiding it - no.

It works out of the box on Gentoo linux. Why do you want to hide this?

In the documentation, it's already marked as "experimental". No change needed there.

I'm referring to you advertising it to users in the mailing lists, such as in:

• https://groups.google.com/g/sage-devel/c/9ETciH8efTE/m/Ov7lIDU_AQAJ

[mkoeppe]

The 'generic' platform is a venv with all needed up to date PyPI packages.

What are you talking about. One cannot make a venv from a venv.

[dimpase]

This case looked as if the person was trying to use already installed Python packages. So I pointed out at this option, why not?

[dimpase]

anyhow I don't think we need this or other GitHub issues to discuss the freedom of speech :-)

[mkoeppe]

why not?

I already explained why not: We know that it does not work -- except on 2 developers' machines.

discuss the freedom of speech

I did see the smiley there, but there's obviously so much to explain.
It's about responsible behavior in our public forums.

[orlitzky]

It works as intended: it uses the system's site packages if they satisfy the version bounds in install-requires.txt. If there's an incompatibility, it's usually because our version bounds are wrong. When this happens on Gentoo, I investigate and fix it. When it happens on another distro that I don't use, I don't. I'm already playing distro developer for one more distro (Sage) than I want to be playing developer for.

There was a user on the mailing list a little bit ago who pointed out a bug with system/SPKG overrides and I'm sure there are other small issues. But largely it does what it says and when it fails it's only pointing out the bad news.

[mkoeppe]

it does what it says and when it fails it's only pointing out the bad news.

My point is that looping our user base into this activity is a bad idea. If anyone wants to fix it for other platforms (I'm not saying that you need to fix it! @orlitzky) , it can already be done using our existing platform testing facilities -- much more effectively. We do NOT need the user input for this.

[mkoeppe]

--enable-system-site-packages is an experimental feature -- I don't think this issue should be a "blocker".

[kwankyu]

"blocker" label is getting overloaded too much.

Let's use "blocker" only for PRs and use "critical" for issues.

[dimpase]

--enable-system-site-packages is an experimental feature -- I don't think this issue should be a "blocker".

an alternative to this feature would be a configure option making all Python packages into pip packages, and (optionally or not) removes any upper bounds on package versions in install-requires.txt.

One may name this option --liberate-python-packages :-)

[orlitzky]

--enable-system-site-packages is an experimental feature -- I don't think this issue should be a "blocker".

an alternative to this feature would be a configure option making all Python packages into pip packages, and (optionally or not) removes any upper bounds on package versions in install-requires.txt.

@tobiasdiez's meson branch "fixes" this and most other problems I have with sage-the-distro by making them irrelevant. After all, --enable-system-site-packages exists only to undo something that sage-the-distro forces on me. If sagelib becomes the first-class citizen and if sage-the-distro is built around it, these issues all go away.

[dimpase]

it's a big if - given that @mkoeppe has blocked me on GitHub, it's going to be "fun"...

[mkoeppe]

Removing "blocker" as explained.