Broken backward compatibility with Juliacall

[Brusa99]

The new version 1.10.1 release broke the package juliacall on Linux systems.
When importing the package from python a segmentation fault occurs.

How to reproduce:

1. Have julia version 1.10.1 installed in your system (or no version at all, since juliacall installs julia if not present).
2. create a clean conda environment with python (3.9 or superior tested).
3. run pip install juliacall
4. open python interactive session and run import juliacall

Note: this does not happen with version 1.10.0 and on Mac devices.

[vchuravy]

Can you post the actual backtrace you are seeing?

[LilithHafner]

I cannot reproduce this on asahi linux

x@fedora:~/Downloads$ python
Python 3.12.1 (main, Dec 18 2023, 00:00:00) [GCC 13.2.1 20231205 (Red Hat 13.2.1-6)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import juliacall
[juliapkg] Locating Julia ^1.6.1
[juliapkg] Using Julia 1.10.1 at /home/x/.julia/juliaup/julia-1.10.1+0.aarch64.linux.gnu/bin/julia
[juliapkg] Using Julia project at /home/x/.julia/environments/pyjuliapkg
[juliapkg] Installing packages:
           julia> import Pkg
           julia> Pkg.add([Pkg.PackageSpec(name="PythonCall", uuid="6099a3de-0909-46bc-b1f4-468b9a2dfc0d")])
           julia> Pkg.resolve()
           julia> Pkg.precompile()
   Resolving package versions...
    Updating `~/.julia/environments/pyjuliapkg/Project.toml`
  [6099a3de] + PythonCall v0.9.15
    Updating `~/.julia/environments/pyjuliapkg/Manifest.toml`
  [992eb4ea] + CondaPkg v0.2.22
  [9a962f9c] + DataAPI v1.16.0
  [e2d170a0] + DataValueInterfaces v1.0.0
  [82899510] + IteratorInterfaceExtensions v1.0.0
  [692b3bcd] + JLLWrappers v1.5.0
  [0f8b85d8] + JSON3 v1.14.0
  [1914dd2f] + MacroTools v0.5.13
  [0b3b1443] + MicroMamba v0.1.14
  [bac558e1] + OrderedCollections v1.6.3
  [69de0a69] + Parsers v2.8.1
  [fa939f87] + Pidfile v1.3.0
  [aea7be01] + PrecompileTools v1.2.0
  [21216c6a] + Preferences v1.4.1
  [6099a3de] + PythonCall v0.9.15
  [ae029012] + Requires v1.3.0
  [6c6a2e73] + Scratch v1.2.1
  [856f2bd8] + StructTypes v1.10.0
  [3783bdb8] + TableTraits v1.0.1
  [bd369af6] + Tables v1.11.1
  [e17b2a0c] + UnsafePointers v1.0.0
  [f8abcde7] + micromamba_jll v1.4.9+0
  [0dad84c5] + ArgTools v1.1.1
  [56f22d72] + Artifacts
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [f43a241f] + Downloads v1.6.0
  [7b1f6079] + FileWatching
  [b77e0a4c] + InteractiveUtils
  [4af54fe1] + LazyArtifacts
  [b27032c2] + LibCURL v0.6.4
  [76f85450] + LibGit2
  [8f399da3] + Libdl
  [37e2e46d] + LinearAlgebra
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [a63ad114] + Mmap
  [ca575930] + NetworkOptions v1.2.0
  [44cfe95a] + Pkg v1.10.0
  [de0858da] + Printf
  [3fa0cd96] + REPL
  [9a3f8284] + Random
  [ea8e919c] + SHA v0.7.0
  [9e88b42a] + Serialization
  [6462fe0b] + Sockets
  [fa267f1f] + TOML v1.0.3
  [a4e569a6] + Tar v1.10.0
  [8dfed614] + Test
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode
  [e66e0078] + CompilerSupportLibraries_jll v1.1.0+0
  [deac9b47] + LibCURL_jll v8.4.0+0
  [e37daf67] + LibGit2_jll v1.6.4+0
  [29816b5a] + LibSSH2_jll v1.11.0+1
  [c8ffd9c3] + MbedTLS_jll v2.28.2+1
  [14a3606d] + MozillaCACerts_jll v2023.1.10
  [4536629a] + OpenBLAS_jll v0.3.23+4
  [83775a58] + Zlib_jll v1.2.13+1
  [8e850b90] + libblastrampoline_jll v5.8.0+1
  [8e850ede] + nghttp2_jll v1.52.0+1
  [3f19e933] + p7zip_jll v17.4.0+2
Precompiling project...
  6 dependencies successfully precompiled in 38 seconds. 19 already precompiled.
  1 dependency had output during precompilation:
┌ micromamba_jll
│   Downloading artifact: micromamba
│  
│  [pid 71524] waiting for IO to finish:
│   Handle type        uv_handle_t->data
│   timer              0x40401550->0xfffec4500ac0
│  This means that a package has started a background task or event source that has not finished running. For precompilation to complete successfully, the event source needs to be closed explicitly. See the developer documentation on fixing precompilation hangs for more help.
│  
│  [pid 71524] waiting for IO to finish:
│   Handle type        uv_handle_t->data
│   timer              0x40401550->0xfffec4500ac0
│  This means that a package has started a background task or event source that has not finished running. For precompilation to complete successfully, the event source needs to be closed explicitly. See the developer documentation on fixing precompilation hangs for more help.
└  
  No Changes to `~/.julia/environments/pyjuliapkg/Project.toml`
  No Changes to `~/.julia/environments/pyjuliapkg/Manifest.toml`
>>> juliacall.Main.VERSION
Julia: v"1.10.1"
>>> juliacall.Main.versioninfo()
Julia Version 1.10.1
Commit 7790d6f0641 (2024-02-13 20:41 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (aarch64-linux-gnu)
  CPU: 8 × unknown
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, generic)
Threads: 1 default, 0 interactive, 1 GC (on 8 virtual cores)
Environment:
  JULIA_PYTHONCALL_EXE = /usr/bin/python

[Brusa99]

Here is what I get when I try to reproduce the error:

Python 3.10.13 (main, Sep 11 2023, 13:44:35) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import juliacall
[juliapkg] Locating Julia ^1.6.1
[juliapkg] Installing Julia 1.10.1 using JuliaUp
[juliapkg] Querying Julia versions from https://julialang-s3.julialang.org/bin/versions.json
[juliapkg] WARNING: About to install Julia 1.10.1 to /home/brusa/miniconda3/envs/reproduce/julia_env/pyjuliapkg/install.
[juliapkg]   If you use juliapkg in more than one environment, you are likely to have Julia
[juliapkg]   installed in multiple locations. It is recommended to install JuliaUp
[juliapkg]   (https://github.com/JuliaLang/juliaup) or Julia (https://julialang.org/downloads)
[juliapkg]   yourself.
[juliapkg] Downloading Julia from https://julialang-s3.julialang.org/bin/linux/x64/1.10/julia-1.10.1-linux-x86_64.tar.gz
             downloaded 87.5 MB of 160.1 MB
             download complete
[juliapkg] Verifying download
[juliapkg] Installing Julia 1.10.1 to /home/brusa/miniconda3/envs/reproduce/julia_env/pyjuliapkg/install
[juliapkg] Using Julia 1.10.1 at /home/brusa/miniconda3/envs/reproduce/julia_env/pyjuliapkg/install/bin/julia
[juliapkg] Using Julia project at /home/brusa/miniconda3/envs/reproduce/julia_env
[juliapkg] Installing packages:
           julia> import Pkg
           julia> Pkg.add([Pkg.PackageSpec(name="PythonCall", uuid="6099a3de-0909-46bc-b1f4-468b9a2dfc0d")])
           julia> Pkg.resolve()
           julia> Pkg.precompile()
   Resolving package versions...
    Updating `~/miniconda3/envs/reproduce/julia_env/Project.toml`
  [6099a3de] + PythonCall v0.9.15
    Updating `~/miniconda3/envs/reproduce/julia_env/Manifest.toml`
  [992eb4ea] + CondaPkg v0.2.22
  [9a962f9c] + DataAPI v1.16.0
  [e2d170a0] + DataValueInterfaces v1.0.0
  [82899510] + IteratorInterfaceExtensions v1.0.0
  [692b3bcd] + JLLWrappers v1.5.0
  [0f8b85d8] + JSON3 v1.14.0
  [1914dd2f] + MacroTools v0.5.13
  [0b3b1443] + MicroMamba v0.1.14
  [bac558e1] + OrderedCollections v1.6.3
  [69de0a69] + Parsers v2.8.1
  [fa939f87] + Pidfile v1.3.0
  [aea7be01] + PrecompileTools v1.2.0
  [21216c6a] + Preferences v1.4.1
  [6099a3de] + PythonCall v0.9.15
  [ae029012] + Requires v1.3.0
  [6c6a2e73] + Scratch v1.2.1
  [856f2bd8] + StructTypes v1.10.0
  [3783bdb8] + TableTraits v1.0.1
  [bd369af6] + Tables v1.11.1
  [e17b2a0c] + UnsafePointers v1.0.0
  [f8abcde7] + micromamba_jll v1.4.9+0
  [0dad84c5] + ArgTools v1.1.1
  [56f22d72] + Artifacts
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [f43a241f] + Downloads v1.6.0
  [7b1f6079] + FileWatching
  [b77e0a4c] + InteractiveUtils
  [4af54fe1] + LazyArtifacts
  [b27032c2] + LibCURL v0.6.4
  [76f85450] + LibGit2
  [8f399da3] + Libdl
  [37e2e46d] + LinearAlgebra
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [a63ad114] + Mmap
  [ca575930] + NetworkOptions v1.2.0
  [44cfe95a] + Pkg v1.10.0
  [de0858da] + Printf
  [3fa0cd96] + REPL
  [9a3f8284] + Random
  [ea8e919c] + SHA v0.7.0
  [9e88b42a] + Serialization
  [6462fe0b] + Sockets
  [fa267f1f] + TOML v1.0.3
  [a4e569a6] + Tar v1.10.0
  [8dfed614] + Test
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode
  [e66e0078] + CompilerSupportLibraries_jll v1.1.0+0
  [deac9b47] + LibCURL_jll v8.4.0+0
  [e37daf67] + LibGit2_jll v1.6.4+0
  [29816b5a] + LibSSH2_jll v1.11.0+1
  [c8ffd9c3] + MbedTLS_jll v2.28.2+1
  [14a3606d] + MozillaCACerts_jll v2023.1.10
  [4536629a] + OpenBLAS_jll v0.3.23+4
  [83775a58] + Zlib_jll v1.2.13+1
  [8e850b90] + libblastrampoline_jll v5.8.0+1
  [8e850ede] + nghttp2_jll v1.52.0+1
  [3f19e933] + p7zip_jll v17.4.0+2
Precompiling project...
  1 dependency successfully precompiled in 2 seconds. 24 already precompiled.
  No Changes to `~/miniconda3/envs/reproduce/julia_env/Project.toml`
  No Changes to `~/miniconda3/envs/reproduce/julia_env/Manifest.toml`
zsh: segmentation fault (core dumped)  python

I have been able to reproduce it also on a Ubuntu VM.
Both devices have no GPU, here is the output of lscpu on my device:

Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         48 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  8
  On-line CPU(s) list:   0-7
Vendor ID:               AuthenticAMD
  Model name:            AMD Ryzen 3 5300U with Radeon Graphics
    CPU family:          23
    Model:               104
    Thread(s) per core:  2
    Core(s) per socket:  4
    Socket(s):           1
    Stepping:            1
    Frequency boost:     enabled
    CPU(s) scaling MHz:  47%
    CPU max MHz:         3900,0000
    CPU min MHz:         1400,0000
    BogoMIPS:            5192,50
    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpu
                         id extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dn
                         owprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rd
                         t_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd cppc arat npt lbrv svm_lock nri
                         p_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca
Virtualization features:
  Virtualization:        AMD-V
Caches (sum of all):
  L1d:                   128 KiB (4 instances)
  L1i:                   128 KiB (4 instances)
  L2:                    2 MiB (4 instances)
  L3:                    4 MiB (1 instance)
NUMA:
  NUMA node(s):          1
  NUMA node0 CPU(s):     0-7
Vulnerabilities:
  Gather data sampling:  Not affected
  Itlb multihit:         Not affected
  L1tf:                  Not affected
  Mds:                   Not affected
  Meltdown:              Not affected
  Mmio stale data:       Not affected
  Retbleed:              Mitigation; untrained return thunk; SMT enabled with STIBP protection
  Spec rstack overflow:  Mitigation; safe RET
  Spec store bypass:     Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1:            Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:            Mitigation; Retpolines, IBPB conditional, STIBP always-on, RSB filling, PBRSB-eIBRS Not affected
  Srbds:                 Not affected
  Tsx async abort:       Not affected

[LilithHafner]

This feels similar to JuliaPy/PythonCall.jl#215, but you have a clean environment, so it shouldn't be a pytorch-julia incompatibility.

[topolarity]

@gbaraldi and I have run into a similar issue upon loading libgomp (which CompilerSupportLibraries_jll` loads) from within Python with RTLD_DEEPBIND. The reproducer looks something like this:

// test.c
#include <stdio.h>
extern char **environ;

__attribute__((constructor))
static void my_init() {
    printf("%p \n", environ[0]); // libgomp expects that `environ` has been initialized by libc
}

# test.py
import ctypes, os
lib = ctypes.CDLL("./libtest.so", os.RTLD_DEEPBIND) # loading Julia-provided libgomp also segfaults

$ clang test.c -shared -o libtest.so
$ python3 test.py # segfaults for some versions of Python
Segmentation fault (core dumped)

The problem is that the environ symbol Python provides (if built without -fPIC) conflicts with the definition provided by glibc, and the resolved symbol is incorrect if loading with RTLD_DEEPBIND:

$ readelf -sW  /usr/bin/python3 | grep environ
  1622: 0000000000a92620     8 OBJECT  WEAK   DEFAULT   29 environ@GLIBC_2.2.5 (2)
  1700: 0000000000a92620     8 OBJECT  GLOBAL DEFAULT   29 __environ@GLIBC_2.2.5 (2)
$ readelf -sW /usr/lib/x86_64-linux-gnu/libc.so.6 | grep environ
   133: 0000000000222200     8 OBJECT  WEAK   DEFAULT   35 _environ@@GLIBC_2.2.5
   724: 0000000000222200     8 OBJECT  GLOBAL DEFAULT   35 __environ@@GLIBC_2.2.5
   958: 0000000000222200     8 OBJECT  WEAK   DEFAULT   35 environ@@GLIBC_2.2.5

This wasn't an issue before 1.10.1 because libgomp did not use environ until it was updated to 13.2.0