Code loading can not find stdlibs outside the sysimage

[fredrikekre]

$ cd $(mktemp -d)

$ pkg --project=. add Tar Zlib_jll
   Resolving package versions...
    Updating `/tmp/tmp.bB2fl6JXHs/Project.toml`
  [a4e569a6] + Tar
  [83775a58] + Zlib_jll
    Updating `/tmp/tmp.bB2fl6JXHs/Manifest.toml`
  [0dad84c5] + ArgTools
  [8f399da3] + Libdl
  [ea8e919c] + SHA
  [a4e569a6] + Tar
  [83775a58] + Zlib_jll

$ JULIA_LOAD_PATH=$PWD/Project.toml julia -e 'using Zlib_jll'
ERROR: ArgumentError: Package Zlib_jll [83775a58-1f1d-513f-b197-d71354ab007a] is required but does not seem to be installed:
 - Run `Pkg.instantiate()` to install all recorded dependencies.

Stacktrace:
 [1] _require(pkg::Base.PkgId)
   @ Base ./loading.jl:986
 [2] require(uuidkey::Base.PkgId)
   @ Base ./loading.jl:910
 [3] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:897

$ JULIA_LOAD_PATH=$PWD/Project.toml julia -e 'using Tar'

$

[KristofferC]

The difference is about if the stdlib is in the sysimage or not. Since @stdlib is not in LOAD_PATH and the package is not in the sysimage, we cannot locate the package. Not sure what can be done about this.

[fredrikekre]

So previously all stdlibs have been in the sysimage?

[KristofferC]

Yes

[fredrikekre]

I assumed code loading had logic to find them anyway, just like Pkg does. I guess we need to implement that then? I think that this is a pretty common pattern to run in an isolated env and use

JULIA_LOAD_PATH=$PWD/Project.toml

I use that alot at least and I have recommended it on several occasion in various forums etc.
It is also a regression since on 1.5 you can selectively make stdlibs available by putting the relevant ones in your project, but now you have to have all or nothing (with @stdlib in path).

[staticfloat]

IMO this is working as intended. If you want to be able to load stdlibs, you should have @stdlib in your Julia load path. If we always manually add @stdlib onto the end of JULIA_LOAD_PATH, that makes it so that purposefully removing it has no effect, which is (in my mind) a worse regression than what we currently have.

What use cases are there for setting JULIA_LOAD_PATH? I've never done that before.

[fredrikekre]

If you want to be able to load stdlibs, you should have @stdlib in your Julia load path.

I disagree, it should be enough to have them in your project + manifest. What is the point of pkg> adding stdlibs if thats not the case?

If we always manually add @stdlib onto the end of JULIA_LOAD_PATH, that makes it so that purposefully removing it has no effect, which is (in my mind) a worse regression than what we currently have.

Thats not what I am suggesting. The previous behavior was that with JULIA_LOAD_PATH=$PWD/Project.toml you could load what was in [deps] and only what was in [deps]. You could not load LinearAlgebra if you hadn't added it for example.

What use cases are there for setting JULIA_LOAD_PATH? I've never done that before.

It is useful if you want to have control over the exact env and don't rely on stuff lower in LOAD_PATH by accident.

[KristofferC]

I don't think we can fix this for 1.6. It is also quite rare to load _jlls in toplevel scripts. But we need to figure out a better way long term.

[fredrikekre]

I believe this is enough but I haven't had time to test it properly yet:

diff --git a/base/loading.jl b/base/loading.jl
index 093e14e..f6fafd8 100644
--- a/base/loading.jl
+++ b/base/loading.jl
@@ -290,6 +290,10 @@ function locate_package(pkg::PkgId)::Union{Nothing,String}
             path = manifest_uuid_path(env, pkg)
             path === nothing || return entry_path(path, pkg.name)
         end
+        # Allow loading of stdlibs if the name/uuid are given
+        # e.g. if they have been explicitly added to the project/manifest
+        path = manifest_uuid_path(Sys.STDLIB, pkg)
+        path === nothing || return entry_path(path, pkg.name)
     end
     return nothing
 end