sagemath-standard: some fixes in build dependencies

[tornaria]

• [build-system] requires: neither jupyter_core nor ppl are needed to build sagemath-standard.
• networkx: remove upper bound since 3.3 works just fine.

📝 Checklist

• The title is concise and informative.
• The description explains in detail what this PR is about.

[tornaria]

For the record, I switched void-linux package to pypi sdist and it's working fine with networkx 3.3 and it doesn't use jupyter_core or pplpy as build time dependencies (it does have them as runtime dependencies so it will install them for testing).

[kiwifb]

I am fine with removing the limit to networkx-3.3, but instead of dropping the upper limit completely we could make it 3.4 as a measure of future proofing just in case.
I would like some real guidance (point me to documentation) on whether the stuff in pyproject.yml is just build time or if it needs to includes runtime. I thought those were needed to get a working package not just built it.

[github-actions]

Documentation preview for this PR (built with commit 556ebe6; changes) is ready! 🎉
This preview will update shortly after each push to this PR.

[tornaria]

I am fine with removing the limit to networkx-3.3, but instead of dropping the upper limit completely we could make it 3.4 as a measure of future proofing just in case.

And then we have to patch at every version update, which is more inconvenient. In fact, very few packages have an upper bound

$ grep "<" build/pkgs/*/version_requirements.txt
build/pkgs/cython/version_requirements.txt:cython >=3.0, != 3.0.3, <4.0
build/pkgs/exceptiongroup/version_requirements.txt:exceptiongroup; python_version<"3.11"
build/pkgs/importlib_metadata/version_requirements.txt:importlib_metadata >=4.13; python_version<"3.11"
build/pkgs/importlib_resources/version_requirements.txt:importlib_resources >= 5.7; python_version<"3.11"
build/pkgs/sphinx/version_requirements.txt:sphinx >=5.2, <8
build/pkgs/sympy/version_requirements.txt:sympy >=1.6, <2.0
build/pkgs/typing_extensions/version_requirements.txt:typing_extensions >= 4.4.0; python_version<'3.11'

At least the other three cases have an upper bound given as a major version change (cython<4, sphinx<8 and sympy<2).

I would like some real guidance (point me to documentation) on whether the stuff in pyproject.yml is just build time or if it needs to includes runtime. I thought those were needed to get a working package not just built it.

I just tried to use pip to build from pypi using all system packages and it did build networkx because it didn't like my networkx-3.3. So I guess this is a "runtime" dependency, but it triggered to me when building.

Indeed, without this patch, my package reports this error:

$ pip check | grep sagemath
sagemath-standard 10.4b4 has requirement networkx<3.3,>=2.4, but you have networkx 3.3.

[dimpase]

lgtm

[dimpase]

I am fine with removing the limit to networkx-3.3, but instead of dropping the upper limit completely we could make it 3.4 as a measure of future proofing just in case.

that's rather a measure of creating more makework. I'd rather see no upper limit at all.

[mkoeppe]

At least the other three cases have an upper bound given as a major version change

That's not relevant because networkx does not distinguish major vs. minor versions in its deprecation policy:
https://networkx.org/documentation/stable/developer/deprecations.html

[mkoeppe]

@dcoudert Any opinion on future-compatibility of our use of networkx in sage.graphs?

[mkoeppe]

I would like some real guidance (point me to documentation) on whether the stuff in pyproject.yml is just build time or if it needs to includes runtime. I thought those were needed to get a working package not just built it.

@kiwifb It's correct that what is declared in pyproject.toml's [build-system] requires is what is needed at the time of running setup.py (for building either metadata or the sdist or the wheel).

This is the separation of phases per PEP517/518/660.

Per this separation of phases, ordinarily, we would normally not even have to look whether anything in the Sage library imports from pplpy or jupyter at the build time. We would just check that there is no cimport (there isn't).

However, we are currently breaking the separation of phases by putting "." in sys.path in setup.py
https://github.com/sagemath/sage/blob/develop/pkgs/sagemath-standard/setup.py#L37

That some "jupyter*" appeared in these dependencies at all can probably be attributed to the installation of the kernel spec in https://github.com/sagemath/sage/blob/develop/src/sage_setup/command/sage_install.py#L28 via sage.repl.ipython_kernel.install (one of the violations of the separation of phases). https://github.com/sagemath/sage/blob/develop/src/sage/repl/ipython_kernel/install.py#L188 does not, but conceivably should use the correct jupyter facility for installing the spec instead of just guessing paths: see

• Rewrite jupyter kernel installation code using KernelSpecManager, create kernel install script (entry point) #30298

(Note that the poorly motivated re-integration of sage_setup in sagemath-standard in #37287 has made the violation of separation of phases worse.)

[dcoudert]

@dcoudert Any opinion on future-compatibility of our use of networkx in sage.graphs?

I have not followed the recent developments of networkx (new functions, deprecations, etc.) but the roadmap does not indicate major changes and mention the will to improve interoperability with other tools.

If needed, we can reduce our dependency to networkx (deprecate use of networkx when we have reliable and faster methods, code some methods instead of calling the networkx one and paying a transformation of the data structure, etc.).

[dimpase]

The troubles with Jupyter kernel are due to vendoring Jupyter in sage distro. If it was just an optional pip package, we'd have figured out proper ways of installing the kernel years ago.

[mkoeppe]

The troubles with Jupyter kernel

What are you talking about. There's no "trouble with the Jupyter kernel".

[tornaria]

About the jupyter kernel install:

• in the past jupyter_core.paths.ENV_JUPYTER_PATH[0] was used for the location to install the sagemath kernel
• this was changed in june 2018 (ad0ee29)
• the current behaviour matches what, e.g., ipykernel does (https://github.com/ipython/ipykernel/blob/main/pyproject.toml#L83)
• it's similar for nbextensions and labextensions (e.g., https://github.com/ipython/ipyparallel/blob/main/pyproject.toml#L105)

I don't see as a big deal if there's a reason to build-depend on jupyter_core, but I usually try to minimize build dependencies and I've been building without jupyter_core forever. Same about pplpy.

OTOH, jinja2 is necessary for the interpreters autogen (which is run in build_wheel, not in bootstrap). I mention this because I saw a PR somewhere removing jinja2, but it does not work.

[tornaria]

Also note: when the wheel is built, there's no information on the install system. The only point in time when it would make sense to use jupyter_core to decide the install location is when installing the wheel but one cannot run code at this step.

Thus, the only reasonable way is to install the files in DATA/share/jupyter/kernels/sagemath/ (this would actually be in sagemath_standard-10.4b4.data/data/share/jupyter/kernels/sagemath/ in the wheel; the installer knows to place sagemath_standard-10.4b4.data/data/some/path in $PREFIX/some/path)

[kiwifb]

I'll say that jupyter_core was not a bother to me. I did figure out that we do not use it anymore for kernel install and using jupyter is clearly an optional thing. But pplpy would be needed at runtime even if it is not at build time. Which is why I asked about whether the dependencies in pyproject.toml are supposed to cover runtime.

The comment on the file does suggest build time is all that is required but when you install something with pip, pip needs to know about runtime dependencies if you want the install to be functional. And that's the only place pip can pull runtime dependency as far as I can see.

[mkoeppe]

We should also provide a script for users to install the kernel into a different environment, see #30298.

[mkoeppe]

pip needs to know about runtime dependencies

Yes, that is currently done in setup.cfg (install-requires), which is to be moved to a different section of pyproject.toml

• pkgs/sagemath-standard: Move metadata from setup.cfg to pyproject.toml #37902

[kiwifb]

pip needs to know about runtime dependencies

Yes, that is currently done in setup.cfg (install-requires), which is to be moved to a different section of pyproject.toml

• pkgs/sagemath-standard: Move metadata from setup.cfg to pyproject.toml #37902

Thanks for the clarification.

[mkoeppe]

OTOH, jinja2 is necessary for the interpreters autogen (which is run in build_wheel, not in bootstrap).

Yes,

I mention this because I saw a PR somewhere removing jinja2

No, we removed it from the required dependencies of sage-setup because most of the modularized distributions that use sage-setup do not need it. We made it an optional dependency (extra-requires), which is then added by those few distributions that need it.

[mkoeppe]

This broke build/pkgs/pplpy/version_requirements.txt, build/pkgs/jupyter_core/version_requirements.txt (generated files since #36982)

[mkoeppe]

I'm fixing this as part of #37902.