Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Current master always re-precompiles REPLExt when entering pkg>-mode in a clean environment #53264

Closed
Seelengrab opened this issue Feb 9, 2024 · 12 comments · Fixed by #53314
Closed
Milestone

Comments

@Seelengrab
Copy link
Contributor

Steps:

  • Build current master (95233619745ef84c96af82299b162aa008a7debd)
  • rm your default environment completely
  • Start the REPL (julia -q)
  • enter pkg> mode
Log from my machine with `JULIA_DEBUG=loading`
[sukera@tower ~]$ JULIA_DEBUG=loading julia -q
┌ Debug: Loading object cache file /home/sukera/julia/usr/share/julia/compiled/v1.11/Base64/D7K0n_QHftB.so for Base64 [2a0f44e3-6c83-55bd-87e4-b1978d98bd5f]
└ @ Base loading.jl:1127
┌ Debug: Loading object cache file /home/sukera/julia/usr/share/julia/compiled/v1.11/Markdown/AREjX_QHftB.so for Markdown [d6f4376e-aef5-505a-96c1-9c027394607a]
└ @ Base loading.jl:1127
┌ Debug: Loading object cache file /home/sukera/julia/usr/share/julia/compiled/v1.11/InteractiveUtils/0TrXF_QHftB.so for InteractiveUtils [b77e0a4c-d291-57a0-90e8-8db25a27a240]
└ @ Base loading.jl:1127
┌ Debug: Loading object cache file /home/sukera/julia/usr/share/julia/compiled/v1.11/StyledStrings/UcVoM_QHftB.so for StyledStrings [f489334b-da3d-4c2e-b8f0-e476e12c162b]
└ @ Base loading.jl:1127
┌ Debug: Loading object cache file /home/sukera/julia/usr/share/julia/compiled/v1.11/Unicode/E4Hzs_QHftB.so for Unicode [4ec0a83e-493e-50e2-b9ac-8f72acf5a8f5]
└ @ Base loading.jl:1127
┌ Debug: Loading object cache file /home/sukera/julia/usr/share/julia/compiled/v1.11/REPL/u0gqU_QHftB.so for REPL [3fa0cd96-eef1-5676-8a61-b3b8758bbffb]
└ @ Base loading.jl:1127
julia> ┌ Debug: Loading object cache file /home/sukera/julia/usr/share/julia/compiled/v1.11/Printf/3FQLY_QHftB.so for Printf [de0858da-6303-5e67-8744-51eddeeeb8d7]
└ @ Base loading.jl:1127
┌ Debug: Loading object cache file /home/sukera/julia/usr/share/julia/compiled/v1.11/Dates/p8See_QHftB.so for Dates [ade2ca70-3891-5945-98fb-dc099432e06a]
└ @ Base loading.jl:1127
┌ Debug: Loading object cache file /home/sukera/julia/usr/share/julia/compiled/v1.11/TOML/mjrwE_QHftB.so for TOML [fa267f1f-6049-4f14-aa54-33bafae1ed76]
└ @ Base loading.jl:1127
┌ Debug: Loading object cache file /home/sukera/julia/usr/share/julia/compiled/v1.11/NetworkOptions/J8H6s_QHftB.so for NetworkOptions [ca575930-c2e3-43a9-ace4-1e988b2c1908]
└ @ Base loading.jl:1127
┌ Debug: Loading object cache file /home/sukera/julia/usr/share/julia/compiled/v1.11/MbedTLS_jll/u5NEn_QHftB.so for MbedTLS_jll [c8ffd9c3-330d-5841-b78e-0817d7145fa1]
└ @ Base loading.jl:1127
┌ Debug: Loading object cache file /home/sukera/julia/usr/share/julia/compiled/v1.11/LibSSH2_jll/K6mup_QHftB.so for LibSSH2_jll [29816b5a-b9ab-546f-933c-edad1886dfa8]
└ @ Base loading.jl:1127
┌ Debug: Loading object cache file /home/sukera/julia/usr/share/julia/compiled/v1.11/LibGit2_jll/nfCpg_QHftB.so for LibGit2_jll [e37daf67-58a4-590a-8e99-b0245dd2ffc5]
└ @ Base loading.jl:1127
┌ Debug: Loading object cache file /home/sukera/julia/usr/share/julia/compiled/v1.11/LibGit2/xrYJZ_QHftB.so for LibGit2 [76f85450-5226-5b5a-8eaa-529ad045b433]
└ @ Base loading.jl:1127
┌ Debug: Loading object cache file /home/sukera/julia/usr/share/julia/compiled/v1.11/ArgTools/aGHFV_QHftB.so for ArgTools [0dad84c5-d112-42e6-8d28-ef12dabb789f]
└ @ Base loading.jl:1127
┌ Debug: Loading object cache file /home/sukera/julia/usr/share/julia/compiled/v1.11/nghttp2_jll/KTGSA_QHftB.so for nghttp2_jll [8e850ede-7688-5339-a07c-302acd2aaf8d]
└ @ Base loading.jl:1127
┌ Debug: Loading object cache file /home/sukera/julia/usr/share/julia/compiled/v1.11/LibCURL_jll/9JWaY_QHftB.so for LibCURL_jll [deac9b47-8bc7-5906-a0fe-35ac56dc84c0]
└ @ Base loading.jl:1127
┌ Debug: Loading object cache file /home/sukera/julia/usr/share/julia/compiled/v1.11/MozillaCACerts_jll/XKIUi_QHftB.so for MozillaCACerts_jll [14a3606d-f60d-562e-9121-12d972cd8159]
└ @ Base loading.jl:1127
┌ Debug: Loading object cache file /home/sukera/julia/usr/share/julia/compiled/v1.11/LibCURL/ht49g_QHftB.so for LibCURL [b27032c2-a3e7-50c8-80cd-2d36dbcbfd21]
└ @ Base loading.jl:1127
┌ Debug: Loading object cache file /home/sukera/julia/usr/share/julia/compiled/v1.11/Downloads/eiA4B_QHftB.so for Downloads [f43a241f-c20a-4ad4-852c-f6b1247861c6]
└ @ Base loading.jl:1127
┌ Debug: Loading object cache file /home/sukera/julia/usr/share/julia/compiled/v1.11/Tar/G9ZYP_QHftB.so for Tar [a4e569a6-e804-4fa4-b0f3-eef7a1d5b13e]
└ @ Base loading.jl:1127
┌ Debug: Loading object cache file /home/sukera/julia/usr/share/julia/compiled/v1.11/p7zip_jll/dfuGM_QHftB.so for p7zip_jll [3f19e933-33d8-53b3-aaab-bd5110c3b7a0]
└ @ Base loading.jl:1127
┌ Debug: Loading object cache file /home/sukera/julia/usr/share/julia/compiled/v1.11/UUIDs/SIw1t_QHftB.so for UUIDs [cf7118a7-6976-5b1a-9a39-7adc72f591a4]
└ @ Base loading.jl:1127
┌ Debug: Loading object cache file /home/sukera/julia/usr/share/julia/compiled/v1.11/Logging/PWFjL_QHftB.so for Logging [56ddb016-857b-54e1-b83d-db4d58db5568]
└ @ Base loading.jl:1127
┌ Debug: Loading object cache file /home/sukera/julia/usr/share/julia/compiled/v1.11/Serialization/zGad9_QHftB.so for Serialization [9e88b42a-f829-5b0c-bbe9-9e923198166b]
└ @ Base loading.jl:1127
┌ Debug: Loading object cache file /home/sukera/julia/usr/share/julia/compiled/v1.11/Pkg/tUTdb_QHftB.so for Pkg [44cfe95a-1eb2-52ea-b672-e2afdf69b78f]
└ @ Base loading.jl:1127
┌ Debug: Rejecting cache file /home/sukera/.julia/compiled/v1.11/REPLExt/m7ggP_oYtoC.ji because uuid mapping for Base.PkgId(Base.UUID("e5eb5ef1-03cf-53a7-ae1d-5a66b08e832b"), "REPLExt") => Base.PkgId(Base.UUID("d6f4376e-aef5-505a-96c1-9c027394607a"), "Markdown") has changed, expected Base.PkgId(Base.UUID("e5eb5ef1-03cf-53a7-ae1d-5a66b08e832b"), "REPLExt") => nothing
└ @ Base loading.jl:3457
┌ Debug: Rejecting cache file /home/sukera/julia/usr/share/julia/compiled/v1.11/REPLExt/m7ggP_QHftB.ji because uuid mapping for Base.PkgId(Base.UUID("e5eb5ef1-03cf-53a7-ae1d-5a66b08e832b"), "REPLExt") => Base.PkgId(Base.UUID("d6f4376e-aef5-505a-96c1-9c027394607a"), "Markdown") has changed, expected Base.PkgId(Base.UUID("e5eb5ef1-03cf-53a7-ae1d-5a66b08e832b"), "REPLExt") => nothing
└ @ Base loading.jl:3457
┌ Debug: Rejecting cache file /home/sukera/julia/usr/share/julia/compiled/v1.11/REPLExt/m7ggP_nPPjv.ji for Base.PkgId(Base.UUID("e5eb5ef1-03cf-53a7-ae1d-5a66b08e832b"), "REPLExt") since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = 0, inline = true, opt_level = 2
│   cache file:      use_pkgimages = true, debug_level = 1, check_bounds = 1, inline = true, opt_level = 2
└ @ Base loading.jl:3347
┌ Debug: Rejecting cache file /home/sukera/.julia/compiled/v1.11/REPLExt/m7ggP_oYtoC.ji because uuid mapping for Base.PkgId(Base.UUID("e5eb5ef1-03cf-53a7-ae1d-5a66b08e832b"), "REPLExt") => Base.PkgId(Base.UUID("d6f4376e-aef5-505a-96c1-9c027394607a"), "Markdown") has changed, expected Base.PkgId(Base.UUID("e5eb5ef1-03cf-53a7-ae1d-5a66b08e832b"), "REPLExt") => nothing
└ @ Base loading.jl:3457
┌ Debug: Rejecting cache file /home/sukera/julia/usr/share/julia/compiled/v1.11/REPLExt/m7ggP_QHftB.ji because uuid mapping for Base.PkgId(Base.UUID("e5eb5ef1-03cf-53a7-ae1d-5a66b08e832b"), "REPLExt") => Base.PkgId(Base.UUID("d6f4376e-aef5-505a-96c1-9c027394607a"), "Markdown") has changed, expected Base.PkgId(Base.UUID("e5eb5ef1-03cf-53a7-ae1d-5a66b08e832b"), "REPLExt") => nothing
└ @ Base loading.jl:3457
┌ Debug: Rejecting cache file /home/sukera/julia/usr/share/julia/compiled/v1.11/REPLExt/m7ggP_nPPjv.ji for Base.PkgId(Base.UUID("e5eb5ef1-03cf-53a7-ae1d-5a66b08e832b"), "REPLExt") since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = 0, inline = true, opt_level = 2
│   cache file:      use_pkgimages = true, debug_level = 1, check_bounds = 1, inline = true, opt_level = 2
└ @ Base loading.jl:3347
┌ Debug: Rejecting cache file /home/sukera/.julia/compiled/v1.11/REPLExt/m7ggP_oYtoC.ji because uuid mapping for Base.PkgId(Base.UUID("e5eb5ef1-03cf-53a7-ae1d-5a66b08e832b"), "REPLExt") => Base.PkgId(Base.UUID("d6f4376e-aef5-505a-96c1-9c027394607a"), "Markdown") has changed, expected Base.PkgId(Base.UUID("e5eb5ef1-03cf-53a7-ae1d-5a66b08e832b"), "REPLExt") => nothing
└ @ Base loading.jl:3457
┌ Debug: Rejecting cache file /home/sukera/julia/usr/share/julia/compiled/v1.11/REPLExt/m7ggP_QHftB.ji because uuid mapping for Base.PkgId(Base.UUID("e5eb5ef1-03cf-53a7-ae1d-5a66b08e832b"), "REPLExt") => Base.PkgId(Base.UUID("d6f4376e-aef5-505a-96c1-9c027394607a"), "Markdown") has changed, expected Base.PkgId(Base.UUID("e5eb5ef1-03cf-53a7-ae1d-5a66b08e832b"), "REPLExt") => nothing
└ @ Base loading.jl:3457
┌ Debug: Rejecting cache file /home/sukera/julia/usr/share/julia/compiled/v1.11/REPLExt/m7ggP_nPPjv.ji for Base.PkgId(Base.UUID("e5eb5ef1-03cf-53a7-ae1d-5a66b08e832b"), "REPLExt") since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = 0, inline = true, opt_level = 2
│   cache file:      use_pkgimages = true, debug_level = 1, check_bounds = 1, inline = true, opt_level = 2
└ @ Base loading.jl:3347
[ Info: Precompiling REPLExt [e5eb5ef1-03cf-53a7-ae1d-5a66b08e832b] (cache misses: mismatched flags (2), dep uuid changed (4))
┌ Debug: Generating object cache file for REPLExt [e5eb5ef1-03cf-53a7-ae1d-5a66b08e832b]
└ @ Base loading.jl:2486
┌ Debug: Loading object cache file /home/sukera/julia/usr/share/julia/compiled/v1.11/Base64/D7K0n_QHftB.so for Base64 [2a0f44e3-6c83-55bd-87e4-b1978d98bd5f]
└ @ Base loading.jl:1127
┌ Debug: Loading object cache file /home/sukera/julia/usr/share/julia/compiled/v1.11/Markdown/AREjX_QHftB.so for Markdown [d6f4376e-aef5-505a-96c1-9c027394607a]
└ @ Base loading.jl:1127
┌ Debug: Loading object cache file /home/sukera/julia/usr/share/julia/compiled/v1.11/UUIDs/SIw1t_QHftB.so for UUIDs [cf7118a7-6976-5b1a-9a39-7adc72f591a4]
└ @ Base loading.jl:1127
┌ Debug: Loading object cache file /home/sukera/julia/usr/share/julia/compiled/v1.11/Printf/3FQLY_QHftB.so for Printf [de0858da-6303-5e67-8744-51eddeeeb8d7]
└ @ Base loading.jl:1127
┌ Debug: Loading object cache file /home/sukera/julia/usr/share/julia/compiled/v1.11/Dates/p8See_QHftB.so for Dates [ade2ca70-3891-5945-98fb-dc099432e06a]
└ @ Base loading.jl:1127
┌ Debug: Loading object cache file /home/sukera/julia/usr/share/julia/compiled/v1.11/StyledStrings/UcVoM_QHftB.so for StyledStrings [f489334b-da3d-4c2e-b8f0-e476e12c162b]
└ @ Base loading.jl:1127
┌ Debug: Loading object cache file /home/sukera/julia/usr/share/julia/compiled/v1.11/InteractiveUtils/0TrXF_QHftB.so for InteractiveUtils [b77e0a4c-d291-57a0-90e8-8db25a27a240]
└ @ Base loading.jl:1127
┌ Debug: Loading object cache file /home/sukera/julia/usr/share/julia/compiled/v1.11/Unicode/E4Hzs_QHftB.so for Unicode [4ec0a83e-493e-50e2-b9ac-8f72acf5a8f5]
└ @ Base loading.jl:1127
┌ Debug: Loading object cache file /home/sukera/julia/usr/share/julia/compiled/v1.11/REPL/u0gqU_QHftB.so for REPL [3fa0cd96-eef1-5676-8a61-b3b8758bbffb]
└ @ Base loading.jl:1127
┌ Debug: Loading object cache file /home/sukera/julia/usr/share/julia/compiled/v1.11/TOML/mjrwE_QHftB.so for TOML [fa267f1f-6049-4f14-aa54-33bafae1ed76]
└ @ Base loading.jl:1127
┌ Debug: Loading object cache file /home/sukera/julia/usr/share/julia/compiled/v1.11/NetworkOptions/J8H6s_QHftB.so for NetworkOptions [ca575930-c2e3-43a9-ace4-1e988b2c1908]
└ @ Base loading.jl:1127
┌ Debug: Loading object cache file /home/sukera/julia/usr/share/julia/compiled/v1.11/MbedTLS_jll/u5NEn_QHftB.so for MbedTLS_jll [c8ffd9c3-330d-5841-b78e-0817d7145fa1]
└ @ Base loading.jl:1127
┌ Debug: Loading object cache file /home/sukera/julia/usr/share/julia/compiled/v1.11/LibSSH2_jll/K6mup_QHftB.so for LibSSH2_jll [29816b5a-b9ab-546f-933c-edad1886dfa8]
└ @ Base loading.jl:1127
┌ Debug: Loading object cache file /home/sukera/julia/usr/share/julia/compiled/v1.11/LibGit2_jll/nfCpg_QHftB.so for LibGit2_jll [e37daf67-58a4-590a-8e99-b0245dd2ffc5]
└ @ Base loading.jl:1127
┌ Debug: Loading object cache file /home/sukera/julia/usr/share/julia/compiled/v1.11/LibGit2/xrYJZ_QHftB.so for LibGit2 [76f85450-5226-5b5a-8eaa-529ad045b433]
└ @ Base loading.jl:1127
┌ Debug: Loading object cache file /home/sukera/julia/usr/share/julia/compiled/v1.11/ArgTools/aGHFV_QHftB.so for ArgTools [0dad84c5-d112-42e6-8d28-ef12dabb789f]
└ @ Base loading.jl:1127
┌ Debug: Loading object cache file /home/sukera/julia/usr/share/julia/compiled/v1.11/nghttp2_jll/KTGSA_QHftB.so for nghttp2_jll [8e850ede-7688-5339-a07c-302acd2aaf8d]
└ @ Base loading.jl:1127
┌ Debug: Loading object cache file /home/sukera/julia/usr/share/julia/compiled/v1.11/LibCURL_jll/9JWaY_QHftB.so for LibCURL_jll [deac9b47-8bc7-5906-a0fe-35ac56dc84c0]
└ @ Base loading.jl:1127
┌ Debug: Loading object cache file /home/sukera/julia/usr/share/julia/compiled/v1.11/MozillaCACerts_jll/XKIUi_QHftB.so for MozillaCACerts_jll [14a3606d-f60d-562e-9121-12d972cd8159]
└ @ Base loading.jl:1127
┌ Debug: Loading object cache file /home/sukera/julia/usr/share/julia/compiled/v1.11/LibCURL/ht49g_QHftB.so for LibCURL [b27032c2-a3e7-50c8-80cd-2d36dbcbfd21]
└ @ Base loading.jl:1127
┌ Debug: Loading object cache file /home/sukera/julia/usr/share/julia/compiled/v1.11/Downloads/eiA4B_QHftB.so for Downloads [f43a241f-c20a-4ad4-852c-f6b1247861c6]
└ @ Base loading.jl:1127
┌ Debug: Loading object cache file /home/sukera/julia/usr/share/julia/compiled/v1.11/Tar/G9ZYP_QHftB.so for Tar [a4e569a6-e804-4fa4-b0f3-eef7a1d5b13e]
└ @ Base loading.jl:1127
┌ Debug: Loading object cache file /home/sukera/julia/usr/share/julia/compiled/v1.11/p7zip_jll/dfuGM_QHftB.so for p7zip_jll [3f19e933-33d8-53b3-aaab-bd5110c3b7a0]
└ @ Base loading.jl:1127
┌ Debug: Loading object cache file /home/sukera/julia/usr/share/julia/compiled/v1.11/Logging/PWFjL_QHftB.so for Logging [56ddb016-857b-54e1-b83d-db4d58db5568]
└ @ Base loading.jl:1127
┌ Debug: Loading object cache file /home/sukera/julia/usr/share/julia/compiled/v1.11/Serialization/zGad9_QHftB.so for Serialization [9e88b42a-f829-5b0c-bbe9-9e923198166b]
└ @ Base loading.jl:1127
┌ Debug: Loading object cache file /home/sukera/julia/usr/share/julia/compiled/v1.11/Pkg/tUTdb_QHftB.so for Pkg [44cfe95a-1eb2-52ea-b672-e2afdf69b78f]
└ @ Base loading.jl:1127
┌ Debug: Loading object cache file /home/sukera/.julia/compiled/v1.11/REPLExt/m7ggP_oYtoC.so for REPLExt [e5eb5ef1-03cf-53a7-ae1d-5a66b08e832b]
└ @ Base loading.jl:1127
┌ Debug: Extension REPLExt of Pkg loaded
└ @ Base loading.jl:1375
(@v1.11) pkg> 
@Seelengrab
Copy link
Contributor Author

FWIW, I also got this earlier while building julia locally:

    JULIA stdlib/REPLExt.release.image
 │ Attempted to find missing packages in package registries but no registries are installed.
 └ Use package mode to install a registry. `pkg> registry add` will install the default registries.

Unsure whether that message is related.

@Seelengrab
Copy link
Contributor Author

Things that didn't help:

  • nuking compiled/v1.11 with rm -rf
  • nuking environments/v1.11 with rm -rf
  • using --startup=no

@KristofferC
Copy link
Sponsor Member

I wonder if it could have to do with how your manifest looked like when you built Julia. Due to #51479 maybe it cached some bad dependency info at build time.

@Seelengrab
Copy link
Contributor Author

Seelengrab commented Feb 9, 2024

No, that doesn't seem to be it - neither when nuking environments/v1.11 nor compiled/v1.11 before building julia, I still get the precompilation either way.

@ctkelley
Copy link

For the last week at least I have to erase .julia/environments/v1.11/Manifest.toml before I install a new nightly to avoid screenfulls of precompile complaints.

@mkitti
Copy link
Contributor

mkitti commented Feb 11, 2024

┌ Debug: Rejecting cache file /home/mkitti/.julia/compiled/v1.11/REPLExt/m7ggP_iP0Qa.ji because uuid mapping for Base.PkgId(Base.UUID("e5eb5ef1-03cf-53a7-ae1d-5a66b08e832b"), "REPLExt") => Base.PkgId(Base.UUID("d6f4376e-aef5-505a-96c1-9c027394607a"), "Markdown") has changed,
expected Base.PkgId(Base.UUID("e5eb5ef1-03cf-53a7-ae1d-5a66b08e832b"), "REPLExt") => nothing

The expected is wrong here. It is due to Base.identify_package returning nothing as invoked below.

Base.identify_package(
     Base.PkgId(Base.UUID("e5eb5ef1-03cf-53a7-ae1d-5a66b08e832b"), "REPLExt"),
     "Markdown"
)

julia/base/loading.jl

Lines 3459 to 3464 in a3e0b62

pkg = identify_package(modkey, req_modkey.name)
if pkg != req_modkey
@debug "Rejecting cache file $cachefile because uuid mapping for $modkey => $req_modkey has changed, expected $modkey => $(repr("text/plain", pkg))"
record_reason(reasons, "dep uuid changed")
return true
end

julia> out = open("/home/mkitti/.julia/compiled/v1.11/REPLExt/m7ggP_iP0Qa.ji", "r") do io
           checksum = Base.isvalid_cache_header(io)
           Base.parse_cache_header(io, "/home/mkitti/.julia/compiled/v1.11/REPLExt/m7ggP_iP0Qa.ji")
       end;

julia> requires = out[2][3]
5-element Vector{Pair{Base.PkgId, Base.PkgId}}:
 Base.PkgId(Base.UUID("e5eb5ef1-03cf-53a7-ae1d-5a66b08e832b"), "REPLExt") => Base.PkgId(Base.UUID("d6f4376e-aef5-505a-96c1-9c027394607a"), "Markdown")
 Base.PkgId(Base.UUID("e5eb5ef1-03cf-53a7-ae1d-5a66b08e832b"), "REPLExt") => Base.PkgId(Base.UUID("cf7118a7-6976-5b1a-9a39-7adc72f591a4"), "UUIDs")
 Base.PkgId(Base.UUID("e5eb5ef1-03cf-53a7-ae1d-5a66b08e832b"), "REPLExt") => Base.PkgId(Base.UUID("ade2ca70-3891-5945-98fb-dc099432e06a"), "Dates")
 Base.PkgId(Base.UUID("e5eb5ef1-03cf-53a7-ae1d-5a66b08e832b"), "REPLExt") => Base.PkgId(Base.UUID("3fa0cd96-eef1-5676-8a61-b3b8758bbffb"), "REPL")
 Base.PkgId(Base.UUID("e5eb5ef1-03cf-53a7-ae1d-5a66b08e832b"), "REPLExt") => Base.PkgId(Base.UUID("44cfe95a-1eb2-52ea-b672-e2afdf69b78f"), "Pkg")

julia> Base.identify_package(requires[1][1], requires[1][2].name)

julia> Base.identify_package(requires[1][1], requires[1][2].name) |> isnothing
true

julia> Base.identify_package(requires[2][1], requires[2][2].name) |> isnothing
true

julia> Base.identify_package(requires[3][1], requires[3][2].name) |> isnothing
true

julia> Base.identify_package(requires[4][1], requires[4][2].name) |> isnothing
true

julia> Base.identify_package(requires[5][1], requires[5][2].name) |> isnothing
true

@mkitti
Copy link
Contributor

mkitti commented Feb 11, 2024

My current thinking is that Base.identify_package should try to check if the where argument is a package extension and if so try to use the parent package instead.

julia/base/loading.jl

Lines 316 to 318 in 4be67e4

function identify_package_env(where::PkgId, name::String)
cache = LOADING_CACHE[]
if cache !== nothing

 function identify_package_env(where::PkgId, name::String) 
     cache = LOADING_CACHE[] 
     if haskey(EXT_PRIMED, where)
         where = EXT_PRIMED[where]
     end
     if cache !== nothing 
     # ...

@mkitti
Copy link
Contributor

mkitti commented Feb 11, 2024

The issue is that @stdlib is just a directory without any Project.toml or Manifest.toml. We end up using Base.implicit_manifest_deps_get which will not work for REPLExt.

julia> REPLExt = Base.PkgId(Base.UUID("e5eb5ef1-03cf-53a7-ae1d-5a66b08e832b"), "REPLExt")
REPLExt [e5eb5ef1-03cf-53a7-ae1d-5a66b08e832b]

julia> Base.implicit_manifest_deps_get(Base.load_path()[2], REPLExt, "Markdown")

julia> Base.implicit_manifest_deps_get(Base.load_path()[2], REPLExt, "REPL")

@mkitti
Copy link
Contributor

mkitti commented Feb 11, 2024

Minimum Demo

$ JULIA_LOAD_PATH=`mktemp -d`: JULIA_DEBUG=loading ./julia -e "using REPL; using Pkg"
...
┌ Debug: Loading object cache file /root/julia-37ed82b289/share/julia/compiled/v1.11/Pkg/tUTdb_RWfMH.so for Pkg [44cfe95a-1eb2-52ea-b672-e2afdf69b78f]                                                └ @ Base loading.jl:1127                                          ┌ Debug: Rejecting cache file /root/.julia/compiled/v1.11/REPLExt/m7ggP_BLIf8.ji because uuid mapping for Base.PkgId(Base.UUID("e5eb5ef1-03cf-53a7-ae1d-5a66b08e832b"), "REPLExt") => Base.PkgId(Base.UUID("d6f4376e-aef5-505a-96c1-9c027394607a"), "Markdown") has changed, expected Base.PkgId(Base.UUID("e5eb5ef1-03cf-53a7-ae1d-5a66b08e832b"), "REPLExt") => nothing                                   └ @ Base loading.jl:3461                                          ┌ Debug: Rejecting cache file /root/julia-37ed82b289/share/julia/compiled/v1.11/REPLExt/m7ggP_RWfMH.ji because uuid mapping for Base.PkgId(Base.UUID("e5eb5ef1-03cf-53a7-ae1d-5a66b08e832b"), "REPLExt") => Base.PkgId(Base.UUID("d6f4376e-aef5-505a-96c1-9c027394607a"), "Markdown") has changed, expected Base.PkgId(Base.UUID("e5eb5ef1-03cf-53a7-ae1d-5a66b08e832b"), "REPLExt") => nothing             └ @ Base loading.jl:3461                                          ┌ Debug: Rejecting cache file /root/julia-37ed82b289/share/julia/compiled/v1.11/REPLExt/m7ggP_VZYKc.ji for Base.PkgId(Base.UUID("e5eb5ef1-03cf-53a7-ae1d-5a66b08e832b"), "REPLExt") since the flags are mismatched                                                      │   current session: use_pkgimages = true, debug_level = 1, check_bounds = 0, inline = true, opt_level = 2                          │   cache file:      use_pkgimages = true, debug_level = 1, check_bounds = 1, inline = true, opt_level = 2                          └ @ Base loading.jl:3351                                          ┌ Debug: Rejecting cache file /root/.julia/compiled/v1.11/REPLExt/m7ggP_BLIf8.ji because uuid mapping for Base.PkgId(Base.UUID("e5eb5ef1-03cf-53a7-ae1d-5a66b08e832b"), "REPLExt") => Base.PkgId(Base.UUID("d6f4376e-aef5-505a-96c1-9c027394607a"), "Markdown") has changed, expected Base.PkgId(Base.UUID("e5eb5ef1-03cf-53a7-ae1d-5a66b08e832b"), "REPLExt") => nothing
└ @ Base loading.jl:3461
┌ Debug: Rejecting cache file /root/julia-37ed82b289/share/julia/compiled/v1.11/REPLExt/m7ggP_RWfMH.ji because uuid mapping for Base.PkgId(Base.UUID("e5eb5ef1-03cf-53a7-ae1d-5a66b08e832b"), "REPLExt") => Base.PkgId(Base.UUID("d6f4376e-aef5-505a-96c1-9c027394607a"), "Markdown") has changed, expected Base.PkgId(Base.UUID("e5eb5ef1-03cf-53a7-ae1d-5a66b08e832b"), "REPLExt") => nothing             └ @ Base loading.jl:3461                                          ┌ Debug: Rejecting cache file /root/julia-37ed82b289/share/julia/compiled/v1.11/REPLExt/m7ggP_VZYKc.ji for Base.PkgId(Base.UUID("e5eb5ef1-03cf-53a7-ae1d-5a66b08e832b"), "REPLExt") since the flags are mismatched                                                      │   current session: use_pkgimages = true, debug_level = 1, check_bounds = 0, inline = true, opt_level = 2                          │   cache file:      use_pkgimages = true, debug_level = 1, check_bounds = 1, inline = true, opt_level = 2                          └ @ Base loading.jl:3351                                          ┌ Debug: Precompiling REPLExt [e5eb5ef1-03cf-53a7-ae1d-5a66b08e832b] (cache misses: mismatched flags (1), dep uuid changed (2))     └ @ Base loading.jl:2600                                          ┌ Debug: Generating object cache file for REPLExt [e5eb5ef1-03cf-53a7-ae1d-5a66b08e832b]                                            └ @ Base loading.jl:2490
...

User Workaround

julia -e "using Pkg; Pkg.add(\"Pkg\")"

This creates an entry for Pkg and REPL as a weakdep in the default environment manifest.

[[deps.Pkg]]                                                      
deps = ["Artifacts", "Dates", "Downloads", "FileWatching", "LibGit2", "Libdl", "Logging", "Markdown", "Printf", "Random", "SHA", "Serialization", "TOML", "Tar", "UUIDs", "p7zip_jll"]                
uuid = "44cfe95a-1eb2-52ea-b672-e2afdf69b78f"                     
version = "1.11.0"                                                                                                                      
[deps.Pkg.extensions]                                             
REPLExt = "REPL"                                                                                                                    
[deps.Pkg.weakdeps]                                              
 REPL = "3fa0cd96-eef1-5676-8a61-b3b8758bbffb"

Potential Solutions

  1. Turn @stdlib into an explicitly manifested environment. The implicit manifest of @stdlib worked before because all standard library packages were available since all packages were top-level directories. REPLExt is not in the implicit manifest since it is not a top-level directory.

  2. Have extensions fallback to identifying packages in its parent environment. Also resolve weakdeps in parent environment. If REPLExt can resolve all dependencies in Pkg's Project.toml, then cache loading should work.

  3. Revert REPLExt

@KristofferC
Copy link
Sponsor Member

KristofferC commented Feb 11, 2024

If there is some corner case with implicit environments and extensions it shouldn't require much to fix that. I thought #52428 would already have handled this.

I mean, Statistics have had an extension on SparseArrays.jl for quite a while so it isn't like this is something new.

@vchuravy
Copy link
Sponsor Member

I mean, Statistics have had an extension on SparseArrays.jl for quite a while so it isn't like this is something new.

I don't think we ship a precompiled copy of that cache-file though in contrast to the REPLExt.

@mkitti
Copy link
Contributor

mkitti commented Feb 11, 2024

I mean, Statistics have had an extension on SparseArrays.jl for quite a while so it isn't like this is something new.

We appear to also have the same issue with SparseArraysExt. Did something else change recently?

┌ Debug: Loading object cache file /root/julia-37ed82b289/share/julia/compiled/v1.11/SparseArrays/P9ieR_RWfMH.so for SparseArrays [2f01184e-e22b-5df5-ae63-d93ebab69eaf]
└ @ Base loading.jl:1127
┌ Debug: Rejecting cache file /root/.julia/compiled/v1.11/SparseArraysExt/k0lPI_BLIf8.ji because uuid mapping for Base.PkgId(Base.UUID("c399192f-b7ce-5053-9844-b5b6ade3ad52"), "SparseArraysExt") => Base.PkgId(Base.UUID("37e2e46d-f89d-539d-b4ee-838fcccc9c8e"), "LinearAlgebra") has changed, expected Base.PkgId(Base.UUID("c399192f-b7ce-5053-9844-b5b6ade3ad52"), "SparseArraysExt") => nothing
└ @ Base loading.jl:3461
┌ Debug: Rejecting cache file /root/.julia/compiled/v1.11/SparseArraysExt/k0lPI_0narh.ji because uuid mapping for Base.PkgId(Base.UUID("c399192f-b7ce-5053-9844-b5b6ade3ad52"), "SparseArraysExt") => Base.PkgId(Base.UUID("37e2e46d-f89d-539d-b4ee-838fcccc9c8e"), "LinearAlgebra") has changed, expected Base.PkgId(Base.UUID("c399192f-b7ce-5053-9844-b5b6ade3ad52"), "SparseArraysExt") => nothing
└ @ Base loading.jl:3461
┌ Debug: Rejecting cache file /root/.julia/compiled/v1.11/SparseArraysExt/k0lPI_BLIf8.ji because uuid mapping for Base.PkgId(Base.UUID("c399192f-b7ce-5053-9844-b5b6ade3ad52"), "SparseArraysExt") => Base.PkgId(Base.UUID("37e2e46d-f89d-539d-b4ee-838fcccc9c8e"), "LinearAlgebra") has changed, expected Base.PkgId(Base.UUID("c399192f-b7ce-5053-9844-b5b6ade3ad52"), "SparseArraysExt") => nothing
└ @ Base loading.jl:3461
┌ Debug: Rejecting cache file /root/.julia/compiled/v1.11/SparseArraysExt/k0lPI_0narh.ji because uuid mapping for Base.PkgId(Base.UUID("c399192f-b7ce-5053-9844-b5b6ade3ad52"), "SparseArraysExt") => Base.PkgId(Base.UUID("37e2e46d-f89d-539d-b4ee-838fcccc9c8e"), "LinearAlgebra") has changed, expected Base.PkgId(Base.UUID("c399192f-b7ce-5053-9844-b5b6ade3ad52"), "SparseArraysExt") => nothing
└ @ Base loading.jl:3461
┌ Debug: Precompiling SparseArraysExt [c399192f-b7ce-5053-9844-b5b6ade3ad52] (cache misses: dep uuid changed (2))
└ @ Base loading.jl:2600
┌ Debug: Generating object cache file for SparseArraysExt [c399192f-b7ce-5053-9844-b5b6ade3ad52]
└ @ Base loading.jl:2490

@vchuravy vchuravy added this to the 1.11 milestone Feb 12, 2024
KristofferC added a commit that referenced this issue Feb 14, 2024
…3314)

Alternative to #53293

This does kind of the same thing but it does not rely on `EXT_PRIMED` to
identify extensions which I think is a better idea since we don't really
want code lookup to depend on the state of Julia itself imo.

Fixes #53264

-----------------------

Co-authored by: Mark Kittisopikul <`markkitti@fosstodon.org`>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants