Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

test_free_threading.test_racing_iter_extend crash #118561

Closed
colesbury opened this issue May 3, 2024 · 4 comments
Closed

test_free_threading.test_racing_iter_extend crash #118561

colesbury opened this issue May 3, 2024 · 4 comments
Labels
tests Tests in the Lib/test dir topic-free-threading type-bug An unexpected behavior, bug, or error type-crash A hard crash of the interpreter, possibly with a core dump

Comments

@colesbury
Copy link
Contributor

colesbury commented May 3, 2024 •
edited by bedevere-app bot
Loading

Crash report

Seen in one of @swtaarrs's PRs:

test_racing_iter_extend (test.test_free_threading.test_list.TestList.test_racing_iter_extend) ... Fatal Python error: Segmentation fault

https://github.com/python/cpython/actions/runs/8942881107/job/24566423243 (crash and then success)

Logs 
test_racing_iter_extend (test.test_free_threading.test_list.TestList.test_racing_iter_extend) ... Fatal Python error: Segmentation fault

Thread 0x00007f062affe700 (most recent call first):
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_free_threading/test_list.py", line 47 in __init__
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_free_threading/test_list.py", line 56 in writer_func
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/threading.py", line 990 in run
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/threading.py", line 1039 in _bootstrap_inner
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/threading.py", line 1010 in _bootstrap

Thread 0x00007f063affe700 (most recent call first):
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_free_threading/test_list.py", line 63 in reader_func
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/threading.py", line 990 in run
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/threading.py", line 1039 in _bootstrap_inner
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/threading.py", line 1010 in _bootstrap

Thread 0x00007f063bfff700 (most recent call first):
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_free_threading/test_list.py", line 61 in reader_func
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/threading.py", line 990 in run
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/threading.py", line 1039 in _bootstrap_inner
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/threading.py", line 1010 in _bootstrap

Thread 0x00007f0649ffd700 (most recent call first):
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_free_threading/test_list.py", line 63 in reader_func
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/threading.py", line 990 in run
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/threading.py", line 1039 in _bootstrap_inner
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/threading.py", line 1010 in _bootstrap

Thread 0x00007f064affe700 (most recent call first):
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_free_threading/test_list.py", line 63 in reader_func
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/threading.py", line 990 in run
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/threading.py", line 1039 in _bootstrap_inner
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/threading.py", line 1010 in _bootstrap

  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/threading.py", line 990 in run
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/threading.py", line 1039 in _bootstrap_inner
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/threading.py", line 1010 in _bootstrap

Thread 0x00007f0639ffd700 (most recent call first):
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_free_threading/test_list.py", line 62 in reader_func
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/threading.py", line 990 in run
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/threading.py", line 1039 in _bootstrap_inner
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/threading.py", line 1010 in _bootstrap

Thread 0x00007f062bfff700 (most recent call first):
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_free_threading/test_list.py", line 62 in reader_func
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/threading.py", line 990 in run
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/threading.py", line 1039 in _bootstrap_inner
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/threading.py", line 1010 in _bootstrap

Thread 0x00007f06cbf7c280 (most recent call first):
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/threading.py", line 1090 in join
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_free_threading/test_list.py", line 74 in test_racing_iter_extend
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/unittest/case.py", line 606 in _callTestMethod
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/unittest/case.py", line 651 in run
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/unittest/case.py", line 707 in __call__
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/unittest/suite.py", line [122](https://github.com/python/cpython/actions/runs/8942881107/job/24566423243#step:19:123) in run
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/unittest/suite.py", line 84 in __call__
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/unittest/suite.py", line 122 in run
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/unittest/suite.py", line 84 in __call__
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/unittest/suite.py", line 122 in run
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/unittest/suite.py", line 84 in __call__
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/unittest/suite.py", line 122 in run
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/unittest/suite.py", line 84 in __call__
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/unittest/runner.py", line 240 in run
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/libregrtest/single.py", line 57 in _run_suite
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/libregrtest/single.py", line 37 in run_unittest
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/libregrtest/single.py", line [132](https://github.com/python/cpython/actions/runs/8942881107/job/24566423243#step:19:133) in test_func
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/libregrtest/single.py", line 88 in regrtest_runner
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/libregrtest/single.py", line 135 in _load_run_test
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/libregrtest/single.py", line 178 in _runtest_env_changed_exc
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/libregrtest/single.py", line 278 in _runtest
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/libregrtest/single.py", line 309 in run_single_test
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/libregrtest/worker.py", line 77 in worker_process
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/libregrtest/worker.py", line 112 in main
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/libregrtest/worker.py", line 116 in <module>
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/runpy.py", line 88 in _run_code
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/runpy.py", line 198 in _run_module_as_main

cc @DinoV

Linked PRs
@colesbury colesbury added tests Tests in the Lib/test dir type-crash A hard crash of the interpreter, possibly with a core dump topic-free-threading labels May 3, 2024
@colesbury
Copy link
Contributor Author

I can reproduce this on main (26bab42).

The issue is that list_get_item_ref can read uninitialized memory because the allocated ob_item is not zero initialized.

@colesbury colesbury added the type-bug An unexpected behavior, bug, or error label May 7, 2024
@colesbury
Copy link
Contributor Author

In particular the issue is the use of list_preallocate_exact, which does not zero initialize the memory in list_extend_fast (and also in list_extend_iter_lock_held)

colesbury added a commit to colesbury/cpython that referenced this issue May 7, 2024
@colesbury
pythongh-118561: Fix crash involving list.extend in free-threaded build
d5ef753 
The `list_preallocate_exact` function did not zero initialize array
contents. In the free-threaded build, this could expose uninitialized
memory to concurrent readers between the call to
`list_preallocate_exact` and the filling of the array contents with
items.
colesbury added a commit to colesbury/cpython that referenced this issue May 7, 2024
@colesbury
pythongh-118561: Fix crash involving list.extend in free-threaded build
f45dfa8 
The `list_preallocate_exact` function did not zero initialize array
contents. In the free-threaded build, this could expose uninitialized
memory to concurrent readers between the call to
`list_preallocate_exact` and the filling of the array contents with
items.
@bedevere-app bedevere-app bot mentioned this issue May 7, 2024
Merged
colesbury added a commit that referenced this issue May 9, 2024
@colesbury
gh-118561: Fix crash involving list.extend in free-threaded build (#1…
2402715 
…18723)

The `list_preallocate_exact` function did not zero initialize array
contents. In the free-threaded build, this could expose uninitialized
memory to concurrent readers between the call to
`list_preallocate_exact` and the filling of the array contents with
items.
miss-islington pushed a commit to miss-islington/cpython that referenced this issue May 9, 2024
@colesbury @miss-islington
pythongh-118561: Fix crash involving list.extend in free-threaded bui…
d60c33b 
…ld (pythonGH-118723)

The `list_preallocate_exact` function did not zero initialize array
contents. In the free-threaded build, this could expose uninitialized
memory to concurrent readers between the call to
`list_preallocate_exact` and the filling of the array contents with
items.
(cherry picked from commit 2402715)

Co-authored-by: Sam Gross <colesbury@gmail.com>
@bedevere-app bedevere-app bot mentioned this issue May 9, 2024
Merged
colesbury added a commit that referenced this issue May 9, 2024
@miss-islington @colesbury
[3.13] gh-118561: Fix crash involving list.extend in free-threaded bu…
846cfb9 
…ild (GH-118723) (#118863)

The `list_preallocate_exact` function did not zero initialize array
contents. In the free-threaded build, this could expose uninitialized
memory to concurrent readers between the call to
`list_preallocate_exact` and the filling of the array contents with
items.
(cherry picked from commit 2402715)

Co-authored-by: Sam Gross <colesbury@gmail.com>
SonicField pushed a commit to SonicField/cpython that referenced this issue May 10, 2024
@colesbury @SonicField
pythongh-118561: Fix crash involving list.extend in free-threaded bui…
fa80517 
…ld (python#118723)

The `list_preallocate_exact` function did not zero initialize array
contents. In the free-threaded build, this could expose uninitialized
memory to concurrent readers between the call to
`list_preallocate_exact` and the filling of the array contents with
items.
@Eclips4
Copy link
Member

Eclips4 commented May 14, 2024

@colesbury can we close this now?

@colesbury
Copy link
Contributor Author

Yeah, I think so

@JelleZijlstra JelleZijlstra mentioned this issue May 28, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
tests Tests in the Lib/test dir topic-free-threading type-bug An unexpected behavior, bug, or error type-crash A hard crash of the interpreter, possibly with a core dump
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@colesbury @Eclips4 and others