Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Relativize more header imports #36754

Merged
merged 3 commits into from
Dec 14, 2023
Merged

Conversation

tobiasdiez
Copy link
Contributor

Follow-up to #36598, relativizing the remaining header imports.

📝 Checklist

  • The title is concise, informative, and self-explanatory.
  • The description explains in detail what this PR is about.
  • I have linked a relevant issue or discussion.
  • I have created tests covering the changes.
  • I have updated the documentation accordingly.

⌛ Dependencies

@mkoeppe
Copy link
Contributor

mkoeppe commented Nov 23, 2023

@tobiasdiez
Copy link
Contributor Author

Thanks, should be fixed now.

Copy link

github-actions bot commented Dec 6, 2023

Documentation preview for this PR (built with commit 233a4ef; changes) is ready! 🎉

@mkoeppe
Copy link
Contributor

mkoeppe commented Dec 6, 2023

LGTM and also applies cleanly to #35095, which has most of these changes.

The Lint failure is unrelated, fixed in #36822

vbraun pushed a commit to vbraun/sage that referenced this pull request Dec 10, 2023
    
<!-- ^^^^^
Please provide a concise, informative and self-explanatory title.
Don't put issue numbers in there, do this in the PR body below.
For example, instead of "Fixes sagemath#1234" use "Introduce new method to
calculate 1+1"
-->
<!-- Describe your changes here in detail -->

<!-- Why is this change required? What problem does it solve? -->
<!-- If this PR resolves an open issue, please link to it here. For
example "Fixes sagemath#12345". -->
<!-- If your change requires a documentation PR, please link it
appropriately. -->

Follow-up to sagemath#36598, relativizing
the remaining header imports.

### 📝 Checklist

<!-- Put an `x` in all the boxes that apply. -->
<!-- If your change requires a documentation PR, please link it
appropriately -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
<!-- Feel free to remove irrelevant items. -->

- [ ] The title is concise, informative, and self-explanatory.
- [ ] The description explains in detail what this PR is about.
- [ ] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation accordingly.

### ⌛ Dependencies

<!-- List all open PRs that this PR logically depends on
- sagemath#12345: short description why this is a dependency
- sagemath#34567: ...
-->

<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
    
URL: sagemath#36754
Reported by: Tobias Diez
Reviewer(s): Matthias Köppe
@kiwifb
Copy link
Member

kiwifb commented Dec 11, 2023

It is not working for me. I am not quite sure what it would need to work in fact

2023-12-11 19:43:28,114 root INFO x86_64-pc-linux-gnu-gcc -Wsign-compare -DNDEBUG -O2 -pipe -ggdb -UNDEBUG -fPIC -Isage/modular/arithgroup -I/usr/lib/python3.11/site-packages/cysignals -Isage/cpython -I/home/portage/sci-mathematics/sage-9999/work/sagemath-standard-9999 -I/usr/lib/python3.11/site-packages/numpy/core/include -I/usr/include/python3.11 -I/home/portage/sci-mathematics/sage-9999/work/sagemath-standard-9999-python3_11/build/cythonized -I/usr/include/python3.11 -c sage/modular/arithgroup/farey.cpp -o /home/portage/sci-mathematics/sage-9999/work/sagemath-standard-9999-python3_11/build/temp.linux-x86_64-cpython-311/sage/modular/arithgroup/farey.o -fno-strict-aliasing -DCYTHON_CLINE_IN_TRACEBACK=1
sage/modular/arithgroup/farey.cpp:33:10: fatal error: farey_symbol.h: No such file or directory
   33 | #include "farey_symbol.h"
      |          ^~~~~~~~~~~~~~~~
compilation terminated.

A fundamental problem seems that the header was moved from the sdist source to the build directory instead of just copied

tarazed ~ # ls -la /home/portage/sci-mathematics/sage-9999/work/sagemath-standard-9999-python3_11/build/lib.linux-x86_64-cpython-311/sage/modular/arithgroup/
total 1536
drwxr-xr-x  2 portage portage   4096 Dec 11 19:43 .
drwxr-xr-x 15 portage portage   4096 Dec 11 19:43 ..
-rw-r--r--  1 portage portage    985 Dec 11 19:34 all.py
-rwxr-xr-x  1 portage portage 646032 Dec 11 19:43 arithgroup_element.cpython-311-x86_64-linux-gnu.so
-rw-r--r--  1 portage portage  13317 Dec 11 19:34 arithgroup_element.pyx
-rw-r--r--  1 portage portage  46601 Dec 11 19:34 arithgroup_generic.py
-rw-r--r--  1 portage portage  88288 Dec 11 19:34 arithgroup_perm.py
-rwxr-xr-x  1 portage portage 544352 Dec 11 19:43 congroup.cpython-311-x86_64-linux-gnu.so
-rw-r--r--  1 portage portage  11625 Dec 11 19:34 congroup.pyx
-rw-r--r--  1 portage portage  10426 Dec 11 19:34 congroup_gamma.py
-rw-r--r--  1 portage portage  20546 Dec 11 19:34 congroup_gamma0.py
-rw-r--r--  1 portage portage  22290 Dec 11 19:34 congroup_gamma1.py
-rw-r--r--  1 portage portage  47152 Dec 11 19:34 congroup_gammaH.py
-rw-r--r--  1 portage portage  21387 Dec 11 19:34 congroup_generic.py
-rw-r--r--  1 portage portage   7610 Dec 11 19:34 congroup_sl2z.py
-rw-r--r--  1 portage portage   2497 Dec 11 19:36 farey_symbol.h
-rw-r--r--  1 portage portage  35741 Dec 11 19:34 farey_symbol.pyx
-rw-r--r--  1 portage portage  13428 Dec 11 19:34 tests.py
tarazed ~ # ls -la /home/portage/sci-mathematics/sage-9999/work/sagemath-standard-9999/sage/modular/arithgroup/total 420
drwxr-xr-x  2 portage portage  4096 Dec 11 19:35 .
drwxr-xr-x 15 portage portage  4096 Dec 11 19:35 ..
-rw-r--r--  1 portage portage   985 Dec 11 19:34 all.py
-rw-r--r--  1 portage portage 13317 Dec 11 19:34 arithgroup_element.pyx
-rw-r--r--  1 portage portage 46601 Dec 11 19:34 arithgroup_generic.py
-rw-r--r--  1 portage portage 88288 Dec 11 19:34 arithgroup_perm.py
-rw-r--r--  1 portage portage 11625 Dec 11 19:34 congroup.pyx
-rw-r--r--  1 portage portage 10426 Dec 11 19:34 congroup_gamma.py
-rw-r--r--  1 portage portage 20546 Dec 11 19:34 congroup_gamma0.py
-rw-r--r--  1 portage portage 22290 Dec 11 19:34 congroup_gamma1.py
-rw-r--r--  1 portage portage 47152 Dec 11 19:34 congroup_gammaH.py
-rw-r--r--  1 portage portage 21387 Dec 11 19:34 congroup_generic.py
-rw-r--r--  1 portage portage  7610 Dec 11 19:34 congroup_sl2z.py
-rw-r--r--  1 portage portage 31782 Dec 11 19:35 farey.cpp
-rw-r--r--  1 portage portage  5090 Dec 11 19:34 farey.hpp
-rw-r--r--  1 portage portage 35741 Dec 11 19:34 farey_symbol.pyx
-rw-r--r--  1 portage portage   888 Dec 11 19:34 sl2z.cpp
-rw-r--r--  1 portage portage  5037 Dec 11 19:34 sl2z.hpp
-rw-r--r--  1 portage portage 13428 Dec 11 19:34 tests.py

I have no idea why.

@kiwifb
Copy link
Member

kiwifb commented Dec 11, 2023

I had not realised that header was generated. The path to it in the build directory would need a -I... as well for things to work.

@kiwifb
Copy link
Member

kiwifb commented Dec 11, 2023

The only way this work for you, is that you must be cythonising in the source tree instead of the build folder. Otherwise, farey_symbol.h cannot be in the same folder as farey.cpp for the relative inclusion to work.

@vbraun vbraun merged commit a1642b1 into sagemath:develop Dec 14, 2023
14 of 18 checks passed
@mkoeppe mkoeppe added this to the sage-10.3 milestone Dec 14, 2023
@tornaria
Copy link
Contributor

I have the same issue as @kiwifb with 10.3.beta2.

I also can't figure out where are the CI checks for this.

This needs to be reverted!

@tornaria tornaria removed this from the sage-10.3 milestone Dec 14, 2023
@tornaria
Copy link
Contributor

FWIW:

cc -fno-strict-overflow -Wsign-compare -DNDEBUG -g -O3 -Wall -fstack-clash-protection -D_FORTIFY_SOURCE=2 -mtune=generic -O2 -pipe -g -fstack-clash-protection -D_FORTIFY_SOURCE=2 -mtune=generic -O2 -pipe -g -fstack-clash-protection -D_FORTIFY_SOURCE=2 -mtune=generic -O2 -pipe -ffile-prefix-map=/builddir/sagemath-10.3.beta2/pkgs/sagemath-standard=. -fPIC -Isage/modular/arithgroup -I/usr/lib/python3.12/site-packages/cysignals -Isage/cpython -I/builddir/sagemath-10.3.beta2/pkgs/sagemath-standard -I/usr/lib/python3.12/site-packages/numpy/core/include -I/usr/include/python3.12 -Ibuild/cythonized -I/usr/include/python3.12 -c sage/modular/arithgroup/farey.cpp -o build/temp.linux-x86_64-cpython-312/sage/modular/arithgroup/farey.o -fno-strict-aliasing -DCYTHON_CLINE_IN_TRACEBACK=1
sage/modular/arithgroup/farey.cpp:32:10: fatal error: farey_symbol.h: No such file or directory
   32 | #include "farey_symbol.h"
      |          ^~~~~~~~~~~~~~~~
compilation terminated.

The command line adds -Ibuild/cythonized so include paths should be relative to that. I.e. sage/modular/arithgroup/farey_symbol.h instead of farey_symbol.h.

But here we don't know why you are making this change, since here you invoke #36598 and there you claim you "don't know why". What is wrong with the full path?

@kiwifb
Copy link
Member

kiwifb commented Dec 14, 2023

Apparently it is related to the meson work. But the farey_symbol header is the only one for which things do not work in an out of source build. If meson can cope with this, it will have to wait for the switch in my opinion. In the meantime I am patching the latest beta.

@tornaria
Copy link
Contributor

Apparently it is related to the meson work. But the farey_symbol header is the only one for which things do not work in an out of source build. If meson can cope with this, it will have to wait for the switch in my opinion. In the meantime I am patching the latest beta.

The "meson work" should not break building sagemath-standard. Please.

And I don't see how the "meson work" is building sage: sage-the-distro should build sage-the-lib as packages (sagemath-standard, etc) otherwise it's like there is one way for "regular distros" and a different way for "sage the distro". Having special ways to build sage-the-lib for sage-the-distro feels like abusing monopoly power.

Right now the (official?) only way I see to build sagemath-standard is to cd into pkgs/sagemath-standard and "python setup.py build". I think "python -m build" didn't work last time I tried. If sage_setup can be phased out in favour of meson-py building that is pep-517 compatible, that would be great. Standard tools instead of NIH: building the car not reinventing the wheel.

Since I'm already ranting, lets get all the steam out: sage_conf is worse than useless, when I tried "pip install sage_conf" it didn't work because of python 3.12, but also it downloaded the whole sage source code into DOT_SAGE, wtf? I already dislike pip because all the mess it does on my filesystem, but this feels bad even for pip.

TL;DR please don't break sagemath-standard.

@tobiasdiez @mkoeppe

@mkoeppe
Copy link
Contributor

mkoeppe commented Dec 14, 2023

I think "python -m build" didn't work last time I tried.

You will typically need python -m build --no-isolation because you want to bring your own dependencies instead of pulling from PyPI. (In particular, sage-conf -- as documented in https://github.com/sagemath/sage/tree/develop/pkgs/sage-conf#sage_conf-in-downstream-distributions.)

@mkoeppe
Copy link
Contributor

mkoeppe commented Dec 14, 2023

I don't see how the "meson work" is building sage: sage-the-distro should build sage-the-lib as packages (sagemath-standard, etc)

I agree, and that's the plan for meson sketched in

@mkoeppe
Copy link
Contributor

mkoeppe commented Dec 14, 2023

I have the same issue as @kiwifb with 10.3.beta2.

I also can't figure out where are the CI checks for this.

We don't have a CI check for building a wheel of sagemath-standard currently.

We build wheels only of sagemath-objects and sagemath-categories (and some of the tiny optional distributions):

These distributions are not big enough to catch the reported failure in sage.modular.

In #35095 (which currently still has ~50K lines of diff against develop), this is covered by the new distribution sagemath-schemes. Any help with upstreaming this work is very welcome.

@tornaria
Copy link
Contributor

I have the same issue as @kiwifb with 10.3.beta2.
I also can't figure out where are the CI checks for this.

We don't have a CI check for building a wheel of sagemath-standard currently.

This wouldn't be a problem if sage-the-distro would eat its own dogfood. I mean, if sage-the-distro would install sage-the-lib in a standard pythonic way that can be used as an example by other distributions.

As a matter of fact, build/pkgs/sagelib/spkg-install.in doesn't look that crazy, except it's building from src instead of pkgs/sagemath-standard (what is the difference? Why do we have two different ways to build sagelib?)

However, sdh_pip_install --no-build-isolation . in build/pkgs/sagelib/spkg-install.in, is that --no-build-isolation what is making the build process not fail here?

@mkoeppe
Copy link
Contributor

mkoeppe commented Dec 14, 2023

This wouldn't be a problem if sage-the-distro would eat its own dogfood.

It does, just not by default (which is the editable build via src/setup.py).

pkgs/sagemath-standard is used if you do ./configure --disable-editable and/or ./configure --enable-wheels.

@mkoeppe
Copy link
Contributor

mkoeppe commented Dec 14, 2023

However, sdh_pip_install --no-build-isolation . in build/pkgs/sagelib/spkg-install.in, is that --no-build-isolation what is making the build process not fail here?

Yes, that's that's the same as what I wrote above in #36754 (comment) regarding pip -m build --no-isolation.

I think in your downstream packaging context, you'll never want to do anything without --no-build-isolation/--no-isolation.

@mkoeppe
Copy link
Contributor

mkoeppe commented Dec 14, 2023

That there are 2 separate build systems for the Sage library is transitional and will go away when the modularization work is merged.

@mkoeppe
Copy link
Contributor

mkoeppe commented Dec 14, 2023

One of the next steps in this direction:

(help is welcome)

@antonio-rojas
Copy link
Contributor

I think "python -m build" didn't work last time I tried.

You will typically need python -m build --no-isolation because you want to bring your own dependencies instead of pulling from PyPI. (In particular, sage-conf -- as documented in https://github.com/sagemath/sage/tree/develop/pkgs/sage-conf#sage_conf-in-downstream-distributions.)

python -m build works fine for me as long as I only build sagemath-standard. If I try to build some optional packages too, then install fails becuse of conflicting files

FileExistsError: File already exists: /build/sagemath/pkg/sagemath/usr/lib/python3.11/site-packages/sage/cython_debug/interpreter

This interpreter file is included by every optional sagemath-* wheel

@mkoeppe
Copy link
Contributor

mkoeppe commented Dec 16, 2023

This interpreter file is included by every optional sagemath-* wheel

Yes, that's #35725

vbraun pushed a commit to vbraun/sage that referenced this pull request Dec 17, 2023
…ude` for `farey_symbol.h`

    
<!-- ^^^^^
Please provide a concise, informative and self-explanatory title.
Don't put issue numbers in there, do this in the PR body below.
For example, instead of "Fixes sagemath#1234" use "Introduce new method to
calculate 1+1"
-->
<!-- Describe your changes here in detail -->

This reverts one change of sagemath#36754, which broke building sagemath-
standard as reported in
sagemath#36754 (comment)

<!-- Why is this change required? What problem does it solve? -->
<!-- If this PR resolves an open issue, please link to it here. For
example "Fixes sagemath#12345". -->
<!-- If your change requires a documentation PR, please link it
appropriately. -->

### 📝 Checklist

<!-- Put an `x` in all the boxes that apply. -->
<!-- If your change requires a documentation PR, please link it
appropriately -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
<!-- Feel free to remove irrelevant items. -->

- [x] The title is concise, informative, and self-explanatory.
- [ ] The description explains in detail what this PR is about.
- [ ] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation accordingly.

### ⌛ Dependencies

<!-- List all open PRs that this PR logically depends on
- sagemath#12345: short description why this is a dependency
- sagemath#34567: ...
-->

<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
    
URL: sagemath#36883
Reported by: Matthias Köppe
Reviewer(s):
@tobiasdiez tobiasdiez deleted the rel_cython_headers branch December 18, 2023 15:34
@tobiasdiez
Copy link
Contributor Author

Apparently it is related to the meson work. But the farey_symbol header is the only one for which things do not work in an out of source build. If meson can cope with this, it will have to wait for the switch in my opinion. In the meantime I am patching the latest beta.

The "meson work" should not break building sagemath-standard. Please.

I'm very sorry for the introduced bug. How do you run the compilation "out of the build" folder? Do you have any idea why only the farey symbol header makes problems but none of the other relative files?

Since I'm already ranting, lets get all the steam out: sage_conf is worse than useless, when I tried "pip install sage_conf" it didn't work because of python 3.12, but also it downloaded the whole sage source code into DOT_SAGE, wtf? I already dislike pip because all the mess it does on my filesystem, but this feels bad even for pip.

You might be happy to hear that neither sage_conf nor sage_setup is needed with the meson compilation at #36524. In fact, it is completely independent of anything in "sage-the-distro". A simple pip install should just work if you have all required deps installed. In fact, the possibility that sagelib itself can check for the required libraries and configure itself during build time is one of the big advantages of meson. If you have the time, I would appreciate if you could give it a try (and any feedback towards what makes downstream packaging easier is welcome there as well, of course).

@kiwifb
Copy link
Member

kiwifb commented Dec 18, 2023

Apparently it is related to the meson work. But the farey_symbol header is the only one for which things do not work in an out of source build. If meson can cope with this, it will have to wait for the switch in my opinion. In the meantime I am patching the latest beta.

The "meson work" should not break building sagemath-standard. Please.

I'm very sorry for the introduced bug. How do you run the compilation "out of the build" folder? Do you have any idea why only the farey symbol header makes problems but none of the other relative files?

That would be because it is the only header that is produced from a cython file. All the other files are present in the source tree. But farey_symbol.h as the product of a cythonisation is put in the build tree rather than the source tree.

@mkoeppe mkoeppe added this to the sage-10.3 milestone Dec 19, 2023
@tobiasdiez
Copy link
Contributor Author

Thanks, makes sense. I was able to find a workaround for the meson compilation. Sorry for the inconvenience

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants