`--enable-experimental-jit` fails to build: `AssertionError: SHT_NOTE` in 3.13.0b1 #118836

mgorny · 2024-05-09T12:35:07Z

Bug report

Bug description:

When trying to build CPython with --enable-experimental-jit against LLVM 18.1.5, I'm getting the following error:

$ make
python3.13 ./Tools/jit/build.py x86_64-pc-linux-gnu

==========================================================
JIT support for x86_64-pc-linux-gnu is still experimental!
         Please report any issues you encounter.          
==========================================================

  + Exception Group Traceback (most recent call last):
  |   File "/home/mgorny/git/cpython/./Tools/jit/build.py", line 28, in <module>
  |     args.target.build(pathlib.Path.cwd(), comment=comment, force=args.force)
  |     ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/mgorny/git/cpython/Tools/jit/_targets.py", line 214, in build
  |     stencil_groups = asyncio.run(self._build_stencils())
  |   File "/usr/lib/python3.13/asyncio/runners.py", line 194, in run
  |     return runner.run(main)
  |            ~~~~~~~~~~^^^^^^
  |   File "/usr/lib/python3.13/asyncio/runners.py", line 118, in run
  |     return self._loop.run_until_complete(task)
  |            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  |   File "/usr/lib/python3.13/asyncio/base_events.py", line 721, in run_until_complete
  |     return future.result()
  |            ~~~~~~~~~~~~~^^
  |   File "/home/mgorny/git/cpython/Tools/jit/_targets.py", line 189, in _build_stencils
  |     async with asyncio.TaskGroup() as group:
  |     ...<4 lines>...
  |             tasks.append(group.create_task(coro, name=opname))
  |   File "/usr/lib/python3.13/asyncio/taskgroups.py", line 154, in __aexit__
  |     raise me from None
  | ExceptionGroup: unhandled errors in a TaskGroup (2 sub-exceptions)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "/home/mgorny/git/cpython/Tools/jit/_targets.py", line 181, in _compile
    |     return await self._parse(o)
    |            ^^^^^^^^^^^^^^^^^^^^
    |   File "/home/mgorny/git/cpython/Tools/jit/_targets.py", line 89, in _parse
    |     self._handle_section(wrapped_section["Section"], group)
    |     ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/mgorny/git/cpython/Tools/jit/_targets.py", line 349, in _handle_section
    |     assert section_type in {
    |            ^^^^^^^^^^^^^^^^^
    |     ...<5 lines>...
    |     }, section_type
    |     ^
    | AssertionError: SHT_NOTE
    +---------------- 2 ----------------
    | Traceback (most recent call last):
    |   File "/home/mgorny/git/cpython/Tools/jit/_targets.py", line 181, in _compile
    |     return await self._parse(o)
    |            ^^^^^^^^^^^^^^^^^^^^
    |   File "/home/mgorny/git/cpython/Tools/jit/_targets.py", line 89, in _parse
    |     self._handle_section(wrapped_section["Section"], group)
    |     ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/mgorny/git/cpython/Tools/jit/_targets.py", line 349, in _handle_section
    |     assert section_type in {
    |            ^^^^^^^^^^^^^^^^^
    |     ...<5 lines>...
    |     }, section_type
    |     ^
    | AssertionError: SHT_NOTE
    +------------------------------------
Exception ignored in: <function BaseSubprocessTransport.__del__ at 0x7f2fa5251760>
Traceback (most recent call last):
  File "/usr/lib/python3.13/asyncio/base_subprocess.py", line 127, in __del__
  File "/usr/lib/python3.13/asyncio/base_subprocess.py", line 104, in close
  File "/usr/lib/python3.13/asyncio/unix_events.py", line 603, in close
  File "/usr/lib/python3.13/asyncio/unix_events.py", line 627, in _close
  File "/usr/lib/python3.13/asyncio/base_events.py", line 829, in call_soon
  File "/usr/lib/python3.13/asyncio/base_events.py", line 552, in _check_closed
RuntimeError: Event loop is closed
make: *** [Makefile:3015: jit_stencils.h] Error 1

I don't recall which alpha I've tested it last on, but I'm pretty sure it used to work (against LLVM 16). I'm getting the same result on main as of 7c87ce7.

This is Gentoo Linux amd64.

I've reproduced by doing (in git repo):

export PATH=/usr/lib/llvm/18/bin:${PATH}
./configure --enable-experimental-jit
make

Resulting log (70k): python-log.txt

CPython versions tested on:

3.13, CPython main branch

Operating systems tested on:

Linux

Linked PRs

The text was updated successfully, but these errors were encountered:

sobolevn · 2024-05-09T13:14:32Z

cc @brandtbucher

brandtbucher · 2024-05-09T13:26:30Z

Thanks for the report!

@savannahostrowski: I think we don’t care about these sections, so it’s probably just a matter of updating the assert. (There may be other missing sections too.)

brandtbucher · 2024-05-09T13:33:01Z

This assert (and others) were originally added early on to make sure we didn’t miss any “important” sections when parsing. Not sure if it’s too fragile in its current form, or if SHT_NOTE is just a common section type that didn’t come up on the platforms we tested.

savannahostrowski · 2024-05-09T14:03:40Z

Thanks for the report!

Makes sense @brandtbucher, I can take a look later today if you'd like.

savannahostrowski · 2024-05-11T02:03:03Z

Alright, so I've probably spent too long trying to set up a Gentoo dev environment 😅 and I'm going to admit defeat.

@mgorny Would you be able to try adding that statement to the assert and let me know if that resolves the issue?

thesamesam · 2024-05-11T02:08:05Z

I'm happy to help (either now or in future) if you need it, fwiw (and/or can give a Dockerfile where hopefully it's reproducible).

Would either of those help?

savannahostrowski · 2024-05-11T02:10:57Z

Ah! If you have a Dockerfile, that'd be amazing. I tried to write one of my own and got stuck in a loop of masked packages. Then, once I got LLVM to start installing, it ultimately failed...

thesamesam · 2024-05-11T03:02:38Z

Let me know how this goes...

FROM gentoo/stage3

# Disable bits which don't work within Docker.
RUN echo 'FEATURES="-ipc-sandbox -pid-sandbox -network-sandbox -usersandbox -mount-sandbox -sandbox"' | cat >> /etc/portage/make.conf
# Speed things up a bit.
RUN echo 'FEATURES="${FEATURES} parallel-install parallel-fetch -merge-sync"' | cat >> /etc/portage/make.conf
RUN echo 'EMERGE_DEFAULT_OPTS="--binpkg-respect-use=y --getbinpkg=y --autounmask-write --autounmask-continue --autounmask-keep-keywords=y --autounmask-use=y"' | cat >> /etc/portage/make.conf
# XXX: Replace -j$(nproc) with some smaller -jN if you run out of RAM
RUN echo "MAKEOPTS='-j$(nproc) -l$(nproc)'" >> /etc/portage/make.conf

RUN emerge-webrsync --quiet
RUN getuto

RUN echo "*/*" >> /etc/portage/package.accept_keywords/all
# By doing this step first (just the deps, not Python itself), we avoid
# losing all our progress if/when Python fails to build.
RUN USE=jit emerge --verbose --oneshot --onlydeps dev-lang/python:3.13
RUN mkdir -p /etc/portage/patches/dev-lang/python:3.13

# You can put patches in /etc/portage/patches/dev-lang/python:3.13/aaa.patch
# to test them.
CMD USE=jit emerge --verbose --oneshot dev-lang/python:3.13

To build it:

docker build -t gentoo gentoo/
docker run -it gentoo

Building LLVM takes a little while - it needs LLVM 18 for the failure, I think, which isn't yet marked stable in Gentoo so there's no binary packages available for it unfortunately.

I added a note about MAKEOPTS in the Dockerfile -- if you get an issue, try reducing that first.

savannahostrowski · 2024-05-11T03:45:19Z

Thanks for writing the Dockerfile! Unfortunately, I landed in the same place that I did last night when trying to write my own Dockerfile. I keep hitting this error:

857.1  * ERROR: sys-devel/llvm-18.1.5::gentoo failed (compile phase):
857.1  *   ninja -v -j11 -l11 distribution failed
857.1  * 
857.1  * Call stack:
857.1  *     ebuild.sh, line  136:  Called src_compile
857.1  *   environment, line 3924:  Called multilib-minimal_src_compile
857.1  *   environment, line 2731:  Called multilib_foreach_abi 'multilib-minimal_abi_src_compile'
857.1  *   environment, line 2998:  Called multibuild_foreach_variant '_multilib_multibuild_wrapper' 'multilib-minimal_abi_src_compile'
857.1  *   environment, line 2691:  Called _multibuild_run '_multilib_multibuild_wrapper' 'multilib-minimal_abi_src_compile'
857.1  *   environment, line 2689:  Called _multilib_multibuild_wrapper 'multilib-minimal_abi_src_compile'
857.1  *   environment, line  600:  Called multilib-minimal_abi_src_compile
857.1  *   environment, line 2725:  Called multilib_src_compile
857.1  *   environment, line 3218:  Called tc-env_build 'cmake_build' 'distribution'
857.1  *   environment, line 4159:  Called cmake_build 'distribution'
857.1  *   environment, line 1425:  Called eninja 'distribution'
857.1  *   environment, line 1896:  Called die
857.1  * The specific snippet of code:
857.1  *       "$@" || die -n "${*} failed"
857.1  * 
857.1  * If you need support, post the output of `emerge --info '=sys-devel/llvm-18.1.5::gentoo'`,
857.1  * the complete build log and the output of `emerge -pqv '=sys-devel/llvm-18.1.5::gentoo'`.
857.1  * The complete build log is located at '/var/tmp/portage/sys-devel/llvm-18.1.5/temp/build.log'.
857.1  * The ebuild environment file is located at '/var/tmp/portage/sys-devel/llvm-18.1.5/temp/environment'.
857.1  * Working directory: '/var/tmp/portage/sys-devel/llvm-18.1.5/work/llvm_build-.arm64'
857.1  * S: '/var/tmp/portage/sys-devel/llvm-18.1.5/work/llvm'
857.1 
857.1 >>> Failed to emerge sys-devel/llvm-18.1.5, Log file:
857.1 
857.1 >>>  '/var/tmp/portage/sys-devel/llvm-18.1.5/temp/build.log'
857.1  * Messages for package sys-libs/libomp-18.1.5:
857.1  * Unable to find kernel sources at /usr/src/linux
857.1  * Unable to calculate Linux Kernel version for build, attempting to use running version
857.1  * Messages for package sys-devel/llvm-18.1.5:
857.1  * ERROR: sys-devel/llvm-18.1.5::gentoo failed (compile phase):
857.1  *   ninja -v -j11 -l11 distribution failed
857.1  * 
857.1  * Call stack:
857.1  *     ebuild.sh, line  136:  Called src_compile
857.1  *   environment, line 3924:  Called multilib-minimal_src_compile
857.1  *   environment, line 2731:  Called multilib_foreach_abi 'multilib-minimal_abi_src_compile'
857.1  *   environment, line 2998:  Called multibuild_foreach_variant '_multilib_multibuild_wrapper' 'multilib-minimal_abi_src_compile'
857.1  *   environment, line 2691:  Called _multibuild_run '_multilib_multibuild_wrapper' 'multilib-minimal_abi_src_compile'
857.1  *   environment, line 2689:  Called _multilib_multibuild_wrapper 'multilib-minimal_abi_src_compile'
857.1  *   environment, line  600:  Called multilib-minimal_abi_src_compile
857.1  *   environment, line 2725:  Called multilib_src_compile
857.1  *   environment, line 3218:  Called tc-env_build 'cmake_build' 'distribution'
857.1  *   environment, line 4159:  Called cmake_build 'distribution'
857.1  *   environment, line 1425:  Called eninja 'distribution'
857.1  *   environment, line 1896:  Called die
857.1  * The specific snippet of code:
857.1  *       "$@" || die -n "${*} failed"
857.1  * 
857.1  * If you need support, post the output of `emerge --info '=sys-devel/llvm-18.1.5::gentoo'`,
857.1  * the complete build log and the output of `emerge -pqv '=sys-devel/llvm-18.1.5::gentoo'`.
857.1  * The complete build log is located at '/var/tmp/portage/sys-devel/llvm-18.1.5/temp/build.log'.
857.1  * The ebuild environment file is located at '/var/tmp/portage/sys-devel/llvm-18.1.5/temp/environment'.
857.1  * Working directory: '/var/tmp/portage/sys-devel/llvm-18.1.5/work/llvm_build-.arm64'
857.1  * S: '/var/tmp/portage/sys-devel/llvm-18.1.5/work/llvm'
857.1 
857.1 
857.1 
857.1 
857.1 
857.1  * Regenerating GNU info directory index...
857.1  * Processed 104 info files.
857.1 
857.1  * IMPORTANT: 16 news items need reading for repository 'gentoo'.
857.1  * Use eselect news read to view new items.
857.1 
------
Dockerfile:17
--------------------
  15 |     # By doing this step first (just the deps, not Python itself), we avoid
  16 |     # losing all our progress if/when Python fails to build.
  17 | >>> RUN USE=jit emerge --verbose --oneshot --onlydeps dev-lang/python:3.13
  18 |     RUN mkdir -p /etc/portage/patches/dev-lang/python:3.13
  19 |     
--------------------
ERROR: failed to solve: process "/bin/sh -c USE=jit emerge --verbose --oneshot --onlydeps dev-lang/python:3.13" did not complete successfully: exit code: 1```

thesamesam · 2024-05-11T03:50:48Z

My guess is you OOMed (check dmesg?) -- could you try changing the MAKEOPTS line to -j4 or something?

(You want it to be roughly: min(threads_on_your_cpu, RAM/2GB).)

mgorny · 2024-05-11T06:15:39Z

@mgorny Would you be able to try adding that statement to the assert and let me know if that resolves the issue?

I can confirm that CPython built correctly now (i.e. with that assert change). I'm testing some packages too, no regressions so far.

savannahostrowski · 2024-05-11T21:38:05Z

@thesamesam Thanks - I realized that right after sending that reply. That said, this doesn't repro on arm64 and when I try to build the image for amd64, I get an illegal exception (I'm on an M3 Pro).

@mgorny Thanks for taking a look and validating that we build correctly now with that change. If you're able to put up a PR, that'd be much appreciated. Happy to review!

…19000)

…pythonGH-119000) (cherry picked from commit e04cd96) Co-authored-by: Michał Górny <mgorny@gentoo.org>

GH-119020) (cherry picked from commit e04cd96) Co-authored-by: Michał Górny <mgorny@gentoo.org>

savannahostrowski · 2024-05-13T23:28:15Z

Looks like this can be closed now, as #119000 and #119020 have been merged.

Thanks @mgorny and @thesamesam for your help!

…pythonGH-119000)

mgorny added the type-bug An unexpected behavior, bug, or error label May 9, 2024

sobolevn added the build The build process and cross-build label May 9, 2024

brandtbucher self-assigned this May 9, 2024

brandtbucher assigned savannahostrowski and unassigned brandtbucher May 9, 2024

bedevere-app bot mentioned this issue May 13, 2024

gh-118836: Fix JIT build error when SHT_NOTE section is present #119000

Merged

brandtbucher pushed a commit that referenced this issue May 13, 2024

GH-118836: Fix JIT build error when SHT_NOTE section is present (GH-1…

e04cd96

…19000)

miss-islington pushed a commit to miss-islington/cpython that referenced this issue May 13, 2024

pythonGH-118836: Fix JIT build error when SHT_NOTE section is present (…

e257f51

…pythonGH-119000) (cherry picked from commit e04cd96) Co-authored-by: Michał Górny <mgorny@gentoo.org>

bedevere-app bot mentioned this issue May 13, 2024

[3.13] GH-118836: Fix JIT build error when SHT_NOTE section is present (GH-119000) #119020

Merged

brandtbucher pushed a commit that referenced this issue May 13, 2024

[3.13] GH-118836: Fix JIT build error when SHT_NOTE section is present (

29a2f9c

GH-119020) (cherry picked from commit e04cd96) Co-authored-by: Michał Górny <mgorny@gentoo.org>

erlend-aasland closed this as completed May 16, 2024

JelleZijlstra mentioned this issue May 28, 2024

backport a9a74da 3.13 #119642

Closed

brandtbucher added the topic-JIT label May 31, 2024

estyxx pushed a commit to estyxx/cpython that referenced this issue Jul 17, 2024

pythonGH-118836: Fix JIT build error when SHT_NOTE section is present (…

2743976

…pythonGH-119000)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`--enable-experimental-jit` fails to build: `AssertionError: SHT_NOTE` in 3.13.0b1 #118836

`--enable-experimental-jit` fails to build: `AssertionError: SHT_NOTE` in 3.13.0b1 #118836

mgorny commented May 9, 2024 •

edited by bedevere-app bot

Loading

sobolevn commented May 9, 2024

brandtbucher commented May 9, 2024

brandtbucher commented May 9, 2024

savannahostrowski commented May 9, 2024 •

edited

Loading

savannahostrowski commented May 11, 2024

thesamesam commented May 11, 2024 •

edited

Loading

savannahostrowski commented May 11, 2024

thesamesam commented May 11, 2024

savannahostrowski commented May 11, 2024

thesamesam commented May 11, 2024

mgorny commented May 11, 2024 •

edited

Loading

savannahostrowski commented May 11, 2024

savannahostrowski commented May 13, 2024

--enable-experimental-jit fails to build: AssertionError: SHT_NOTE in 3.13.0b1 #118836

--enable-experimental-jit fails to build: AssertionError: SHT_NOTE in 3.13.0b1 #118836

Comments

mgorny commented May 9, 2024 • edited by bedevere-app bot Loading

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

sobolevn commented May 9, 2024

brandtbucher commented May 9, 2024

brandtbucher commented May 9, 2024

savannahostrowski commented May 9, 2024 • edited Loading

savannahostrowski commented May 11, 2024

thesamesam commented May 11, 2024 • edited Loading

savannahostrowski commented May 11, 2024

thesamesam commented May 11, 2024

savannahostrowski commented May 11, 2024

thesamesam commented May 11, 2024

mgorny commented May 11, 2024 • edited Loading

savannahostrowski commented May 11, 2024

savannahostrowski commented May 13, 2024

`--enable-experimental-jit` fails to build: `AssertionError: SHT_NOTE` in 3.13.0b1 #118836

`--enable-experimental-jit` fails to build: `AssertionError: SHT_NOTE` in 3.13.0b1 #118836

mgorny commented May 9, 2024 •

edited by bedevere-app bot

Loading

savannahostrowski commented May 9, 2024 •

edited

Loading

thesamesam commented May 11, 2024 •

edited

Loading

mgorny commented May 11, 2024 •

edited

Loading