Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Segmentation fault with threads #47826

Closed
qiaojunfeng opened this issue Dec 7, 2022 · 5 comments
Closed

Segmentation fault with threads #47826

qiaojunfeng opened this issue Dec 7, 2022 · 5 comments
Labels
bug Indicates an unexpected problem or unintended behavior multithreading Base.Threads and related functionality regression Regression in behavior compared to a previous version
Milestone
1.10

Comments

@qiaojunfeng
Copy link

For a simple multithreading code, it seems with version 1.9.0-alpha1, when running with more than 1 threads, julia crash randomly (more than 50%, but there are few chances the code finished fine). However, it is working fine with 1.8.3.

  1. The output of versioninfo()

    julia> versioninfo()
Julia Version 1.9.0-alpha1
Commit 0540f9d7394 (2022-11-15 14:37 UTC)
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 20 × Intel(R) Core(TM) i9-9900X CPU @ 3.50GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-14.0.6 (ORCJIT, skylake-avx512)
  Threads: 1 on 20 virtual cores

  2. How you installed Julia

    juliaup add 1.9  # 1.9.0-alpha1+0.x64.linux.gnu
juliaup add 1.8  # 1.8.3+0.x64.linux.gnu

  3. A minimal working example (MWE), also known as a minimum reproducible example

    nw = 8
nk = 1000
A = zeros(ComplexF64, nw, nw, nk)
B = rand(ComplexF64, nw, nw, nk)

@time Threads.@threads for ik in 1:nk
    for j in 1:nk
        A[:, :, ik] += B[:, :, j]
    end
end

The crash output is somewhat not unique, following are the output I collected:

[3544842] signal (11.1): Segmentation fault
in expression starting at /tmp/test_perf.jl:11
to_indices at ./indices.jl:354 [inlined]
to_indices at ./indices.jl:344 [inlined]
getindex at ./abstractarray.jl:1294
unknown function (ip: 0x7f97b610ef65)
_jl_invoke at /cache/build/default-amdci5-1/julialang/julia-release-1-dot-9/src/gf.c:2450 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-1/julialang/julia-release-1-dot-9/src/gf.c:2632
macro expansion at /home/jqiao/git/wannier90_dev/examples/example16-noqe/test_perf.jl:14 [inlined]
#9#threadsfor_fun#3 at ./threadingconstructs.jl:139
#9#threadsfor_fun at ./threadingconstructs.jl:106 [inlined]
#1 at ./threadingconstructs.jl:84
unknown function (ip: 0x7f97b610db6f)
_jl_invoke at /cache/build/default-amdci5-1/julialang/julia-release-1-dot-9/src/gf.c:2450 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-1/julialang/julia-release-1-dot-9/src/gf.c:2632
jl_apply at /cache/build/default-amdci5-1/julialang/julia-release-1-dot-9/src/julia.h:1868 [inlined]
start_task at /cache/build/default-amdci5-1/julialang/julia-release-1-dot-9/src/task.c:1089
Allocations: 11282400 (Pool: 11281162; Big: 1238); GC: 36

[3545031] signal (11.1): Segmentation fault
in expression starting at /tmp/test_perf.jl:11
ijl_get_nth_field at /cache/build/default-amdci5-1/julialang/julia-release-1-dot-9/src/datatype.c:1500
ijl_get_nth_field_checked at /cache/build/default-amdci5-1/julialang/julia-release-1-dot-9/src/datatype.c:1546
getindex at ./tuple.jl:29 [inlined]
to_indices at ./indices.jl:354 [inlined]
to_indices at ./indices.jl:344 [inlined]
getindex at ./abstractarray.jl:1294
unknown function (ip: 0x7f4e8d2bef65)
_jl_invoke at /cache/build/default-amdci5-1/julialang/julia-release-1-dot-9/src/gf.c:2450 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-1/julialang/julia-release-1-dot-9/src/gf.c:2632
macro expansion at /tmp/test_perf.jl:14 [inlined]
#9#threadsfor_fun#3 at ./threadingconstructs.jl:139
#9#threadsfor_fun at ./threadingconstructs.jl:106 [inlined]
#1 at ./threadingconstructs.jl:84
unknown function (ip: 0x7f4e8d2bdb6f)
_jl_invoke at /cache/build/default-amdci5-1/julialang/julia-release-1-dot-9/src/gf.c:2450 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-1/julialang/julia-release-1-dot-9/src/gf.c:2632
jl_apply at /cache/build/default-amdci5-1/julialang/julia-release-1-dot-9/src/julia.h:1868 [inlined]
start_task at /cache/build/default-amdci5-1/julialang/julia-release-1-dot-9/src/task.c:1089
Allocations: 5218944 (Pool: 5217706; Big: 1238); GC: 16

[3547161] signal (11.1): Segmentation fault
in expression starting at /tmp/test_perf.jl:11
jl_field_isatomic at /cache/build/default-amdci5-1/julialang/julia-release-1-dot-9/src/julia.h:1183 [inlined]
ijl_get_nth_field at /cache/build/default-amdci5-1/julialang/julia-release-1-dot-9/src/datatype.c:1507
ijl_get_nth_field_checked at /cache/build/default-amdci5-1/julialang/julia-release-1-dot-9/src/datatype.c:1546
getindex at ./tuple.jl:29 [inlined]
to_indices at ./indices.jl:354 [inlined]
to_indices at ./indices.jl:344 [inlined]
setindex! at ./abstractarray.jl:1397
unknown function (ip: 0x7f4994111259)
_jl_invoke at /cache/build/default-amdci5-1/julialang/julia-release-1-dot-9/src/gf.c:2450 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-1/julialang/julia-release-1-dot-9/src/gf.c:2632
macro expansion at /tmp/test_perf.jl:14 [inlined]
#9#threadsfor_fun#3 at ./threadingconstructs.jl:139
#9#threadsfor_fun at ./threadingconstructs.jl:106 [inlined]
#1 at ./threadingconstructs.jl:84
unknown function (ip: 0x7f499410db6f)
_jl_invoke at /cache/build/default-amdci5-1/julialang/julia-release-1-dot-9/src/gf.c:2450 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-1/julialang/julia-release-1-dot-9/src/gf.c:2632
jl_apply at /cache/build/default-amdci5-1/julialang/julia-release-1-dot-9/src/julia.h:1868 [inlined]
start_task at /cache/build/default-amdci5-1/julialang/julia-release-1-dot-9/src/task.c:1089
Allocations: 6392485 (Pool: 6391247; Big: 1238); GC: 20
@giordano giordano added multithreading Base.Threads and related functionality regression Regression in behavior compared to a previous version labels Dec 8, 2022
@gbaraldi
Copy link
Member

gbaraldi commented Dec 8, 2022

Bisected to 1a7a131 @vchuravy @vtjnash

@KristofferC KristofferC added this to the 1.9 milestone Dec 8, 2022
@gbaraldi
Copy link
Member

gbaraldi commented Dec 8, 2022

I get an assertion failure with a debug build

julia> @time Threads.@threads for ik in 1:nk
           for j in 1:nk
               A[:, :, ik] += B[:, :, j]
           end
       end
Assertion failed: (ly->fielddesc_type == 2), function jl_field_offset, file julia.h, line 1163.

@vtjnash
Copy link
Member

vtjnash commented Dec 13, 2022

Poking at memory in rr, I saw this was allocated in thread 4 by this thread:

julia> descend(getindex, (Array{Base.Complex{Float64}, 3}, Function, Function, Int64), debuginfo=:none)

Then immediately freed (set to NULL) by thread 1 before we get a chance to use it:

Emission is pretty straightforward:

define nonnull {}* @julia_getindex_1682({}* noundef nonnull align 16 dereferenceable(40) %0, {}* noundef nonnull readonly %1, {}* noundef nonnull readonly %2, i64 signext %3) #0 {                                                                
top:                                                                                                                                                                                                                                               
  %I = alloca {}*, align 8                                                                                                                                                                                                                         
...
  %20 = call nonnull {}* ({}* ({}*, {}**, i32)*, {}*, ...) @julia.call({}* ({}*, {}**, i32)* @jl_f_tuple, {}* null, {}* %1, {}* %2, {}* %19)                                                                                                       
...
  fence syncscope("singlethread") seq_cst                                                                                
  %22 = bitcast {}** %ptls to i64**                                                                                                                                                                                                                
  %23 = getelementptr inbounds i64*, i64** %22, i64 2                                                                                                                                                                                              
  %safepoint = load i64*, i64** %23, align 8                                                                             
  %24 = load volatile i64, i64* %safepoint, align 8                                                                      
  fence syncscope("singlethread") seq_cst                                                                                
...
  %70 = load {}*, {}** %I, align 8                                                                                       
  %71 = call {}* @ijl_get_nth_field_checked({}* %70, i64 0)                                                              
...

But we don't end up putting the GC store until the first real call in this function.

@vtjnash vtjnash added the bug Indicates an unexpected problem or unintended behavior label Dec 13, 2022
@vtjnash
Copy link
Member

vtjnash commented Dec 13, 2022

Seems like @vchuravy possibly already meant to fix this #47393

@gbaraldi
Copy link
Member

Why would that fix this? I though that PR just changed the safepoint block to an intrinsic that we lowered later, or would it also impede the safepoint from moving?

@KristofferC KristofferC mentioned this issue Dec 20, 2022
Merged
@KristofferC KristofferC modified the milestones: 1.9, 1.10 Dec 20, 2022
@vtjnash vtjnash closed this as completed Mar 1, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Indicates an unexpected problem or unintended behavior multithreading Base.Threads and related functionality regression Regression in behavior compared to a previous version
Projects
None yet
Milestone
1.10
Development

No branches or pull requests
5 participants
@vtjnash @giordano @KristofferC @qiaojunfeng @gbaraldi