22% faster startup because sysimage is 33% smaller

[PallHaraldsson]

No description provided.

[KristofferC]

There is ongoing work to move out stdlibs from the sysimage, #49130, #49132, #49134, etc. However, this needs to be done one by one and checks against load time and first call latency has to be done. And for some packages like Pkg there needs to be hooks in place to load the package on ] in the REPL, the precompile workload has to be moved into the Pkg repo etc. Removing everything in one step is too coarse.

[PallHaraldsson]

this needs to be done one by one and checks against load time and first call latency has to be done.

First call of what? Most programs do not care about first call latency of anything related to e.g. LibGit2, Tar, Downloads, and well Pkg, the major offender it seems. Do you have an idea of any other big offender to start with excising? Or a likely first call issue? Wouldn't it be offset by the faster startup? Fully? Everything can be precompiled in the stdlibs.

And for some packages like Pkg there needs to be hooks in place to load the package on ] in the REPL

Are you opposed to this if I make it work? You closed the issue on that, implying you don't want it done, at JuliaLang/Pkg.jl#3570 I was looking into what I can excise and still start. I forgot that I disabled the Pkg, I believe everything else still works.

Pkg alone pulls in 17 dependencies of all 27, many of are not needed by general programs (so it's blocking good progress):

Artifacts = "56f22d72-fd6d-98f1-02f0-08ddc0907c33"
Dates = "ade2ca70-3891-5945-98fb-dc099432e06a"
Downloads = "f43a241f-c20a-4ad4-852c-f6b1247861c6"
FileWatching = "7b1f6079-737a-58dc-b8bc-7a2ca5c1b5ee"
LibGit2 = "76f85450-5226-5b5a-8eaa-529ad045b433"
Libdl = "8f399da3-3557-5675-b5ff-fb832c97cbdb"
Logging = "56ddb016-857b-54e1-b83d-db4d58db5568"
Markdown = "d6f4376e-aef5-505a-96c1-9c027394607a"
Printf = "de0858da-6303-5e67-8744-51eddeeeb8d7"
REPL = "3fa0cd96-eef1-5676-8a61-b3b8758bbffb"
Random = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"
SHA = "ea8e919c-243c-51af-8825-aaa63cd721ce"
Serialization = "9e88b42a-f829-5b0c-bbe9-9e923198166b"
TOML = "fa267f1f-6049-4f14-aa54-33bafae1ed76"
Tar = "a4e569a6-e804-4fa4-b0f3-eef7a1d5b13e"
UUIDs = "cf7118a7-6976-5b1a-9a39-7adc72f591a4"
p7zip_jll = "3f19e933-33d8-53b3-aaab-bd5110c3b7a0"

[KristofferC]

First call of what?

First call to a function in that stdlib. For example a Pkg.add. See e.g. https://github.com/JuliaLang/Pkg.jl/pull/3447/files#diff-83c70a88f749a6ea52fa833d216bf35b2f86437d66a91239219691a4999ec4ee and JuliaLang/Pkg.jl#3445

Are you opposed to this if I make it work? You closed the issue on that, at Pkg, implying you don't want it done.

Pkg was moved out of the sysimage but that caused issues (related to precompilation) so that had to be reverted (JuliaLang/Pkg.jl#3442). Moving Pkg.precompile into Base was one possible solution to that issue. But that seems to be a bigger project than anticipated so the next strategy is to fix the issue that caused the precompilation problem. This is also one of the reasons why this should be done more incrementally. Saying it a bit bluntly perhaps but if it was this easy, it would just have been done.

[PallHaraldsson]

First call to a function in that stdlib. For example a Pkg.add.

(You and) I meant "first call [latency]". I.e. that's a bad example to worry about latency. But yes it has to work, and ] for Pkg mode, so reopen this issue I pointed to? Do you have an example of any function you worry about latency wise in any other stdlib (or you think would simply not work)?

[KristofferC]

that's a bad example to worry about latency.

Hmm, I don't understand why wanting Pkg.add to not take a long time the first time you call it is a bad example?

---

Once Pkg is out, that frees up quite a bit of other stdlibs that can be moved out:

#49428
#49426
#49425

(You can also see from those PRs other things that need updating, like various make files and tests).

[PallHaraldsson]

I don't understand why wanting Pkg.add to not take a long time the first time you call it is a bad example?

I mean compiling from source code on demand should be ok, assuming it doesn't take long. I.e. longer ok, but neither has to be if precompiled into its stdlib. But loading the Pkg code (and more e.g. the deprecated Downloads) is worse for all users, which don't use it, at least for benchmarks.

[vchuravy]

Hitting ] needs to work and it needs to be reasonably snappy

[KristofferC]

But loading the Pkg code (and more e.g. the deprecated Downloads) is worse for all users, which don't use it, at least for benchmarks.

To be clear, the goal is indeed to move out Pkg from the sysimage. It just requires the proper ground work.