Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP/RFC: Use CompilePreferences.jl to configure Python per project #835

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

tkf
Copy link
Member

@tkf tkf commented Sep 25, 2020

This PR uses forthcoming CompilePreferences.jl stdlib JuliaLang/julia#37595 to manage libpython setting for each project. The current design is to move deps/build.jl to a separate package PyPreferences.jl that gets a precompile cache for each configuration of libpython (python executable, using conda or not, PyJulia-mode or not, etc.). It is a separate package because we need to have a UI that can be imported even in a broken status (e.g., invalid python path).

Most of the logic lives in https://github.com/tkf/PyPreferences.jl but I thought it might be a good idea to open a PR to have a place to discuss the direction. At this point, this PR is still more or less for me to play with JuliaLang/julia#37595 API to see if it is compatible with the usecase I have in mind. But feedback to the design/implementation is always welcome!

@tkf
Copy link
Member Author

tkf commented Sep 25, 2020

Here is an illustrative session (with a955681 and tkf/PyPreferences.jl@7006f6c). Suppose I have two projects:

shell> cat py37/Project.toml
[deps]
PyCall = "438e738f-606a-5dbb-bf0a-cddfbfd45ab0"
PyPreferences = "cc9521c6-0242-4dda-8d66-c47a9d9eec02"

shell> cat py38/Project.toml
[deps]
PyCall = "438e738f-606a-5dbb-bf0a-cddfbfd45ab0"
PyPreferences = "cc9521c6-0242-4dda-8d66-c47a9d9eec02"

Using PyPreferences to configure Python versions for each projects:

julia> using PyPreferences

(@v1.6) pkg> activate py37
 Activating environment at `~/.julia/dev/_wt/PyCall/preferences/tmp/py37/Project.toml`

julia> PyPreferences.use_system("python3.7")
PyPreferences.Implementations.PythonPreferences("python3.7", false, false)

(py37) pkg> activate py38
 Activating environment at `~/.julia/dev/_wt/PyCall/preferences/tmp/py38/Project.toml`

julia> PyPreferences.use_system("python3.8")
PyPreferences.Implementations.PythonPreferences("python3.8", false, false)

Now these projects contain compile-preferences table:

shell> cat py37/Project.toml

[compile-preferences.cc9521c6-0242-4dda-8d66-c47a9d9eec02]
python = "python3.7"
[deps]
PyCall = "438e738f-606a-5dbb-bf0a-cddfbfd45ab0"
PyPreferences = "cc9521c6-0242-4dda-8d66-c47a9d9eec02"

shell> cat py38/Project.toml

[compile-preferences.cc9521c6-0242-4dda-8d66-c47a9d9eec02]
python = "python3.8"
[deps]
PyCall = "438e738f-606a-5dbb-bf0a-cddfbfd45ab0"
PyPreferences = "cc9521c6-0242-4dda-8d66-c47a9d9eec02"

julia> exit()

These two projects use two different Python versions:

$ ~/repos/watch/_wt/julia/preferences/usr/bin/julia --project=py37
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.6.0-DEV.957 (2020-09-17)
 _/ |\__'_|_|_|\__'_|  |  preferences/e9e4d60854 (fork: 1 commits, 7 days)
|__/                   |

julia> using PyCall; PyCall.pyversion_build
[ Info: Precompiling PyCall [438e738f-606a-5dbb-bf0a-cddfbfd45ab0]
v"3.7.8"

$ ~/repos/watch/_wt/julia/preferences/usr/bin/julia --project=py38
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.6.0-DEV.957 (2020-09-17)
 _/ |\__'_|_|_|\__'_|  |  preferences/e9e4d60854 (fork: 1 commits, 7 days)
|__/                   |

julia> using PyCall; PyCall.pyversion_build
[ Info: Precompiling PyCall [438e738f-606a-5dbb-bf0a-cddfbfd45ab0]
v"3.8.5"

I can use these two projects without re-compiling PyCall:

$ ~/repos/watch/_wt/julia/preferences/usr/bin/julia --project=py37
...

julia> using PyCall; println(pyimport("sys").version)  # no precompilation
3.7.8 (default, Aug  3 2020, 14:06:56)
[GCC 10.1.0]

$ ~/repos/watch/_wt/julia/preferences/usr/bin/julia --project=py38
...

julia> using PyCall; println(pyimport("sys").version)  # no precompilation
3.8.5 (default, Jul 27 2020, 08:42:51)
[GCC 10.1.0]

@stevengj
Copy link
Member

Sounds good; I've been hoping for Pkg support for build preferences for a long time.

@tkf
Copy link
Member Author

tkf commented Sep 27, 2020

I suppose we'd want to have a shared/global preferences persists across Julia and PyCall versions (exactly like ~/.julia/prefs/PyCall) as a fallback of per-project preference, right? The current design of CompilePreferences.jl JuliaLang/julia#37595 does not let us have such kind of preferences storage and I think it means that we need to implement something like ~/.julia/prefs/PyCall as a different mechanism (JuliaLang/julia#37595 (comment)).

I think there is a way out for either directions in JuliaLang/julia#37595 but I thought you might want to know this.

@PhilipVinc
Copy link
Contributor

Hi @tkf, what's the status of this? Is there any agreement on how this should be implemented?
I'd like to revive this and contribute to it because I would find it extremely useful.

@tkf
Copy link
Member Author

tkf commented Nov 5, 2021

I don't know if @stevengj looked into the implementation (as I posted it as a WIP), but I think we agree with the idea around this. The last time I tried, I didn't have time to integrate this into PyJulia.

If you can revive this again, I think it'd be great. It'd be extra awesome if you can tweak PyJulia simultaneously (or make it non-breaking for PyJulia). But otherwise, we probably need to temporarily break PyJulia after PyCall with the preference support is released.

Also, now that Pkg supports multiple packages in a single repository, I think it'd be easier to add lib/PyPreferences package in this repository.

@stevengj
Copy link
Member

stevengj commented Nov 6, 2021

I would definitely like to switch PyCall to use the new Preferences architecture via https://github.com/JuliaPackaging/Preferences.jl … it doesn't seem like it needs a separate package anymore, so this PR should be reworked; might be easier to start a brand-new PR?

@tkf
Copy link
Member Author

tkf commented Nov 7, 2021

I don't mind if this PR is discarded, but I think it'd be better to separate the preference package. This is because it'd be much easier if the user can set the preference without loading PyCall itself. Without a separate package, we'd need to be able to make PyCall loadable in a "broken" state (i.e., cannot find an appropriate libpython). It's technically possible but I'm strongly against this approach since it'd require us to test PyCall twice if we want to be really sure that it works in both states. Furthermore, PyPreferences approach makes libpython discovery code (the scripts in deps) much easier to test, because we can run them in isolation without re-compiling PyCall.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants