Error when compiling non-inlined functions.

[Klafyvel]

Hi,
Thank you for this nice package.

I am the author of MallocDeques.jl, and I am facing an issue that I don't really understand. MallocDeques.jl consists mostly of inlined functions, which compile fine. However, whenever I try to de-inline some of these functions I get errors from StaticCompiler.jl.

Working inlined example:

using StaticTools, StaticCompiler
using MallocDeques

function test()
    md = MallocDeque{Int}()
    push!(md, 1)
    free(md)
end

compile_executable(test, (), "./", filename="test_compilation")

non-Working de-inlined example

using StaticTools, StaticCompiler
using MallocDeques

function Base.push!(d::MallocDeque{T,C}, x::T) where {T,C}
    if isempty(d.rear)
        MallocDeques.front!(d.rear, 1)
        MallocDeques.back!(d.rear, 0)
    end

    if MallocDeques.back(d.rear) < C
        b = MallocDeques.back(d.rear)
        MallocDeques.data!(d.rear, x, b += 1)
        MallocDeques.back!(d.rear, b)
    else
        new_rear = MallocDeques.rear_deque_block(T, C)
        MallocDeques.back!(new_rear, 1)
        MallocDeques.data!(new_rear, x, 1)
        MallocDeques.prev!(new_rear, d.rear)
        MallocDeques.next!(d.rear, new_rear)
        d.rear = new_rear
        d.nblocks += 1
    end
    d.len += 1
    return d
end

function test()
    md = MallocDeque{Int}()
    push!(md, 1)
    free(md)
end

compile_executable(test, (), "./", filename="test_compilation")

The error

/usr/bin/x86_64-linux-gnu-ld : ./test_compilation.o : dans la fonction « gpu_gc_pool_alloc » :
text:(.text+0x170) : référence indéfinie vers « ijl_throw »
clang-13: error: linker command failed with exit code 1 (use -v to see invocation)
ERROR: LoadError: failed process: Process(setenv(`/home/hugo/.julia/artifacts/75ed94858c831a8af1fcfa51515
09c116570ef05/tools/clang ./wrapper.c ./test_compilation.o -o ./test_compilation`,["LSCOLORS=Gxfxcxdxbxeg
edabagacad", "EDITOR=nvim", "PATH=/home/hugo/.julia/artifacts/75ed94858c831a8af1fcfa5151509c116570ef05/to
ols:/home/hugo/.local/bin:/home/hugo/.local/bin:/home/hugo/.julia/juliaup/bin:/usr/local/sbin:/usr/local/
bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin", "QT_ACCESSIBILITY=1",
 "NVIM_LOG_FILE=/home/hugo/.local/state/nvim/log", "VIM=/tmp/.mount_nvimc0jKRv/usr/share/nvim", "LD_LIBRA
RY_PATH=/home/hugo/.julia/juliaup/julia-1.8.5+0.x64.linux.gnu/bin/../lib/julia:/home/hugo/.julia/juliaup/
julia-1.8.5+0.x64.linux.gnu/bin/../lib/julia:/home/hugo/.julia/juliaup/julia-1.8.5+0.x64.linux.gnu/bin/..
/lib", "DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus", "XDG_SESSION_DESKTOP=ubuntu", "APPDIR=/tm
p/.mount_nvimc0jKRv"  …  "JULIA_NUM_THREADS=auto", "ZSH=/home/hugo/.oh-my-zsh", "GNOME_TERMINAL_SERVICE=:
1.3386", "XDG_SESSION_CLASS=user", "PWD=/home/hugo/Documents/code/Norg.jl", "DISPLAY=:0", "GNOME_SETUP_DI
SPLAY=:1", "APPIMAGE=/home/hugo/nvim.appimage", "SSH_AGENT_LAUNCHER=gnome-keyring", "LS_COLORS=rs=0:di=01
;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43
:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31
:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=0
1;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;
31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=0
1;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz
=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:
*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=0
1;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.
mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01
;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc
=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cg
m=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*
.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;
36:*.spx=00;36:*.xspf=00;36:"]), ProcessExited(1)) [1]

Stacktrace:
 [1] pipeline_error
   @ ./process.jl:565 [inlined]
 [2] run(::Cmd; wait::Bool)
   @ Base ./process.jl:480
 [3] run
   @ ./process.jl:477 [inlined]
 [4] generate_executable(f::Function, tt::Type, path::String, name::String, filename::String; cflags::Cmd
, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ StaticCompiler ~/.julia/packages/StaticCompiler/7kg6k/src/StaticCompiler.jl:392
 [5] compile_executable(f::Function, types::Tuple{}, path::String, name::String; filename::String, cflags
::Cmd, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ StaticCompiler ~/.julia/packages/StaticCompiler/7kg6k/src/StaticCompiler.jl:247
 [6] top-level scope
   @ ~/Documents/code/Norg.jl/test.jl:48
in expression starting at /home/hugo/Documents/code/Norg.jl/test.jl:48

Looking at the generated LLVM IR code I can't see any obvious throw call. I'll keep investigating using Cthulhu, but I wondered if you knew about specific edge cases using non-inlined functions?

[brenhinkeller]

Ah good question.. So I could be entirely wrong, but my understanding of it is as follows:

If a function is inlined, any objects allocated on the stack by that function can instead just be allocated in on the stack of whatever bigger function your function has been inlined into, and that is all great. However, if a function is not inlined, its stack will have to be unwound after it returns, and the caller will need somewhere to put the objects returned by the callee. Julia may deal with this by allocating space with the garbage collector (c.f. "boxing"). This means inserting a call to one or more functions from libjulia that typically include terms like "gc" and "alloc" -- in this case gpu_gc_pool_alloc. In any case, We do not have access to the garbage collector and allocator in Static-land, so this call if inserted will cause compilation to fail.

It does so happen that the gpu_gc_pool_alloc function in turn includes calls to ijl_throw, but that is in this case effectively a red herring -- it is the allocation that is the real problem.

In this case, we have a few potential solutions

• Inline all the things!
• Return only llvm types and pointers (may or may not always be enough? not sure..)
• Use the new @device_override interface from StaticCompiler to override and implement our own gc (?!?)

[brenhinkeller]

To test the second option ("Return only llvm types and pointers") you might try having your non-inlined push! function return 0 instead of d -- I'm actually curious now if that'll work

[Klafyvel]

So, I tried returning 0 instead of d, but I still get the same error. So I guess there still are some unwanted allocations. I'll keep investigating!

[Klafyvel]

By commenting parts of the non-inlined functions it seems that the cause of the error is the line that creates a new block (it contains the malloc call).

But the funny thing is the test function compiles if I only put in that exact line:

function test()
    # md = MallocDeque{Int}()
    # push!(md, 1)
    md = MallocDeques.rear_deque_block(Int, 2)
    free(md)
end

compile_executable(test, (), "./", filename="test_compilation")

[Klafyvel]

I got it to work! It was a tricky issue. Your idea of returning 0 was actually right but I did it the wrong way. In the first non-working example I gave, I was redefining Base.push! after MallocDeques.jl already had defined it. That lead to the return type of push! being Any somehow. Fixing the code directly in MallocDeques.jl fixed the issue.

[brenhinkeller]

Hooray!