Fix 7z and zlib problems

[staticfloat]

The 7z executable wasn't getting copied in for Windows because it lacked $(EXE). Additionally, there was some confusion within the buildsystem as far as whether things were named zlib or libz. This is fixed now, and although it doesn't cause confusion to any of the systems we test on, other packages were hitting issues, as reported in #33470 (comment)

It is also reported that packages such as GR.jl call explicitly $(Sys.BINDIR)/7z.exe; I do not think we should continue to support this, it's really bad behavior for us to put things into $(Sys.BINDIR) in general; that's why I moved things to libexec, so we should encourage packages to just do a VERSION check and look in joinpath(Sys.BINDIR, "..", "libexec") instead.

[musm]

Glad to have the fix. Can you explain the purpose of the newly introduced libexec folder? Is it just to hold the 7z executable or is it expected to be populated by other binaries ?

[staticfloat]

libexec is the standard directory that you put binaries which are excecuted by your own binary, but should not pollute the global namespace of binaries for the entire machine. So, for instance, libexec is not put on the PATH. But Julia knows how to find it for its own internal usage.

If we were to, for instance, ship curl alongside Julia in the future, we would it put there as well.

[Petr-Hlavenka]

I don't know if the cleanup of 7z into libexec is worth the slightly breaking change. I see 7z and possibly curl an exception needed in order to to bootstrap Pkg before the artifact machinery is ready to serve. Do you expect anything else to go into libexec? I'd expect that any other executable should be invoked using the artifact framework.

[musm]

@phlavenk I think the advantage of this approach is that it prevents polluting your environment with unnecessary executables and shared libraries. This can especially be an issue if you add Julia's bin folder to your PATH (I've run into this several times with other programs adding everything to path, including bundled executables, which then take over other installed system executables with the same name).

[staticfloat]

I don't know if the cleanup of 7z into libexec is worth the slightly breaking change

I think it is worth it; we need to be good neighbors to not accidentally override other program's 7z executables.

I see 7z and possibly curl an exception needed in order to to bootstrap Pkg before the artifact machinery is ready to serve.

I'd expect that any other executable should be invoked using the artifact framework.

Precisely; we are of the same mind about this.

[Petr-Hlavenka]

Thank you both for explanation, libexec is definitely the right way of dealing with dependencies.

[nalimilan]

It is also reported that packages such as GR.jl call explicitly $(Sys.BINDIR)/7z.exe; I do not think we should continue to support this, it's really bad behavior for us to put things into $(Sys.BINDIR) in general; that's why I moved things to libexec, so we should encourage packages to just do a VERSION check and look in joinpath(Sys.BINDIR, "..", "libexec") instead.

We should really provide Sys.LIBEXECDIR if we expect packages to use that directory. The Makefiles support a libexecdir variable, so there's no guaranty that the folder is called libexec. For some reason, base/build_h.jl contains lots of paths, but libexecdir is missing.

[nalimilan]

BTW, when USE_SYSTEM_P7ZIP=1, this should probably create a symlink to the system's executable, just like we do for libraries? Otherwise we can't recommend users to look for 7z in libexecdir.

[staticfloat]

We can't know where 7z will be on the host system; there is no single install location that is standard. Packages that want to support non-default installations of Julia should probably push Sys.LIBEXECDIR (I like that suggestion, feel free to open a PR and I'll review it) onto the front of ENV["PATH"], then use Sys.which() to search for 7z.

[nalimilan]

Wouldn't pushing libexecdir in front of of PATH go against the idea that this folder shouldn't interfere with other users? Callers could just try $(Sys.LIBEXECDIR)/7z, and use Sys.which only if that fails.

TBH that's really not a priority for me right now (I just came across this on Slack).

[staticfloat]

I don't mean push it system-wide or even Julia-process-wide; I mean something like:

7z_path = withenv("PATH" => string(Sys.LIBEXECDIR, Sys.iswindows() ? ";" : ":", ENV["PATH"]) do
	Sys.which(`7z`)
end

[rapus95]

Now that this is merged, before packages will change to yet another hardcoded path, might the suggestion Sys.LIBEXECDIR of @nalimilan in #33493 (comment) could be provided? Sure, that way 7z would become part of the public API and would need to be provided (maybe via some which->symlink or just by copying always). Or if applicable by specifying to use BinaryProvider.jl
Either way it would be nice if there was some note on what to change the former line to. Hope this makes it into 1.3 as packages will change there.

[staticfloat]

I would be happy to review a Sys.LIBEXECDIR pull request; feel free to open one and ping me.

[musm]

ref #33777

Where I don't do exactly Sys.LIBEXECDIR but return Base.LIBEXECDIR which is relative to Sys.BINDIR

[musm]

@staticfloat do you recall which commit actually moved 7z to the libexec folder (and created that folder)? Is it this PR or an earlier one?

[musm]

EDIT NVM I think I found it.