Set default exeflags to start procs using active project

[chriselrod]

Maybe a breaking change, also probably not the best way to implement this. Someone setting exeflag may find it surprising that doing so suddenly no longer sets the project to be the active one.

However, I think most people will find this to be the expected behavior, so I'd like at least some discussion on this behavior. It can be surprising to addprocs and suddenly have using error or load different versions than from your current session.

Perhaps there already was some discussion on the desired behavior here?

EDIT:
I tested this locally. I'm willing to bet that this won't work when the procs don't share a filesystem, unless they all happen to have a project at the same path.

So maybe I should edit addprocs(np::Integer, kwargs...), having it forward to something like addprocs(manager; exeflags = get(kwargs, :exeflags, "--project=$(Base.active_project())"), kwargs...) instead, to set the exeflags this way).

[fredrikekre]

See #28781

[jpsamaroo]

+1 to this. The fact that we don't already set this by default is a large inconvenience to the average user, who's typically either running purely local processes, or running remote processes on a shared filesystem. The current behavior often creates situations where the desired packages are loaded on worker 1, but all other workers load a different set of packages (with potentially different structures and functions), causing serialization errors or odd bugs, especially when more complicated distributed computing packages (like Dagger.jl) are used.

[ericphanson]

who's typically either running purely local processes, or running remote processes on a shared filesystem

Just to say this fix also works for a setup where the manager and workers all have copies of the same filesystem (but are not actually shared). That might sound unusual but it comes up when you have remote workers + manager all using the same docker image (e.g. with K8sClusterManagers.jl).

[ViralBShah]

I suppose we either accept this or close.

[JeffBezanson]

👍 Can't see a reason not to do this.

[chriselrod]

Good to merge?

[fredrikekre]

I think it should replicate the full load path instead.

[vchuravy]

@alanedelman just ran into this, and the worker processes and the main process had a completely separate environment that caused weird miss-behavior.

@chriselrod can you rebase this and address @fredrikekre comment?

[tkf]

I guess "full load path" means evaluating load_path_setup_code() on the remote workers on startup? It typically contains something like

julia> print(Base.load_path_setup_code())
append!(empty!(Base.DEPOT_PATH), ["/home/arakaki/.julia", "/home/arakaki/opt/julia/julia-1.6.0/local/share/julia", "/home/ar
akaki/opt/julia/julia-1.6.0/share/julia"])
append!(empty!(Base.DL_LOAD_PATH), String[])
append!(empty!(Base.LOAD_PATH), ["/home/arakaki/.julia/environments/v1.6/Project.toml", "/home/arakaki/opt/julia/julia-1.6.0
/share/julia/stdlib/v1.6"])
ENV["JULIA_LOAD_PATH"] = "/home/arakaki/.julia/environments/v1.6/Project.toml:/home/arakaki/opt/julia/julia-1.6.0/share/juli
a/stdlib/v1.6"
Base.ACTIVE_PROJECT[] = nothing

i.e., also replicate paths like DEPOT_PATH too.