rez-bind python & rez-release depends-on-python: should python package be released?

[alexey-pelykh]

It's more of the what-is-by-design question. The situation is as follows:

1. by default, rez binds python (and os, arch, platform) package to local_packages_path
2. when a package that depends on python is released, the process assumes that all dependencies are released as well, thus forcing to release locally-bound python, os, arch, platform
3. rez-pip by default looks for the latest python & pip packages and corresponding versions may not be present locally

Seems like a collision.

Workaround attempt no. 1: Specify the exact version for rez-pip that matches locally-bound python

Won't help since the same version can be released from different arch, producing

The following package conflicts occurred: (arch-AMD64 <--!--> arch-x86_64)

Workaround attempt no. 2: Remove all published python package versions, forcing it to use locally-bound

• would help for rez-pip
• would help with rez env resolving unless --no-local is specified
• will cause issue with rez-release since not all dependencies are published

Questions:

• by design, what should be the intended approach?
• rez-bind is going to be deprecated (so I've heard)?
• should all possible combinations of python-os-arch-platform be published to shared (cross-platform) rez packages repo?

Thanks in advance for the guidance!

cc @nerdvegas

[instinct-vfx]

I am not sure i really understand the problem. Trying to unravel a little.
First of all, yes rez-bind is going to be deprecated as pretty much everything it does can be done with "normal" packages these days.

Secondly, you don't have to locally bind python necessarily. It can be a central package only (it might still wrap a local installation which is sometimes referred to as a "native" package, or it might carry the whole release as payload). There is no real difference between local_packages_path and packages_path except for local_packages_path being the default target for installation (this is also consistent for rez-build and rez-pip, this is not rez-pip specific). The reason is that a build by default is considered local and dev and only using rez-release or rez-pip -r targets your release_packages_path. To re-iterate: Using rez-pip without -r or using rez-build or using rez-bind without -r is intended to release local dev packages. Not production use packages.

Also note that os, arch and platform are a somewhat special case as they are by default configured as implicit packages, so they are implicitly added to every resolve.

rez-pip does not just look for "the latest python and pip packages". It has the following search order (from the docstring):

 1. Search for rezified python matching python version request;
        2. If found, test if pip is present;
        3. If pip is present, use it;
        4. If not present, search for rezified pip (this is for backwards compatibility);
        5. If rezified pip is found, use it;
        6. If not, fall back to rez's python installation.

It does not matter if the desired versions do not exist locally if they exist centrally. They do have to exist "somewhere" of course for this to work.

I think the main misunderstanding here resolves around local_packages_path vs. packages_path. Always treat local_packages_path as dev and basically only for testing and release everything you intent to really use to packages_path. (Not it does not necessarily matter if either of these are local or central. Both is possible technically but i would strongly recommend to use local_packages_path locally and only while developing or testing and make packages_path a central shared repository with production packages that can also be used by everyone else.

You will have to release all combinations of os/arch/platform to a shared location. And this is also true for python (but as it is not an implicit by default that is no different from any other package out there, say a DCC).

If you are starting out i would also suggest to skip rez-bind and not build new packages to be used with bind. If you are on windows, i would also advise not to use a rez-bind created python package as that comes with a bunch of restrictions and pitfalls on windows. For windows specifically i did a live exercise recently with a bunch of people to walk them through creating a tool to create full python rez packages on windows with full payload and turn that into an automatic tool. There sadly is no recording available but i did share the tool. I might follow up with a tutorial, but i can not promise that. You can find the tool here: https://github.com/techartorg/rez_utils/tree/main/rezify_python

[alexey-pelykh]

First of all, yes rez-bind is going to be deprecated as pretty much everything it does can be done with "normal" packages these days.

Thanks for the confirmation!

Secondly, you don't have to locally bind python necessarily.

https://github.com/AcademySoftwareFoundation/rez/blame/master/README.md#L151 would benefit from some updating since it's phrased as a non-optional step, far from being deprecated.

rez-pip does not just look for "the latest python and pip packages".

If the version is not specified, it uses the latest released https://github.com/AcademySoftwareFoundation/rez/blob/master/src/rez/pip.py#L178 no matter if it's resolvable or not.

  --python-version VERSION

                        python version (rez package) to use, *default is
                        latest.* Note that the pip package(s) will be installed
                        with a dependency on python-MAJOR.MINOR.

Yet that's a bit counterintuitive, especially if rez-pip-3.7 form is used that tries to use 3.9 that happens to be the latest python rez-package released for a different platform/os/arch.

You will have to release all combinations of os/arch/platform to a shared location. And this is also true for python (but as it is not an implicit by default that is no different from any other package out there, say a DCC).

Thanks for explicit confirmation, that was part of the not-so-clearly-formed question as well :)

[alexey-pelykh]

Unless the rez-bind alternative works 99.999% of the time, it'll frustrate new users more to remove rez-bind who are just looking to start / try stuff out.

@ColinKennedy indeed, removing rez-bind would increase "does not work out of the box" complains, yet if there were an official alternative like https://github.com/AcademySoftwareFoundation/rez/wiki/Package-Definition-Guide#example-package that "soft-wraps" system python (the one that rez was installed with/running with) - that would be an option?

[alexey-pelykh]

You will have to release all combinations of os/arch/platform to a shared location.

@instinct-vfx but why if they are implicit? rez-env somepackage --no-local gives

requested packages:
somepackage
~platform==osx  (implicit)
~arch==x86_64   (implicit)
~os==osx-12.4   (implicit)

but rez-env os --no-local

06:40:11 ERROR    PackageNotFoundError: Package could not be found: os==osx-12.4

Since there's indeed no os/osx-12.4 in shared location, yet if there's no os package at all - it will fail to resolve somepackage.
That's the expected behavior? Can I use some "catch-all" version of os package since it does not care about version when implicit?

[ColinKennedy]

alternative like https://github.com/AcademySoftwareFoundation/rez/wiki/Package-Definition-Guide#example-package that "soft-wraps" system python (the one that rez was installed with/running with) - that would be an option?

That's essentially what rez-bind does, in a way. The main differences between the link you've sent and rez-bind is that rez-bind will find the system python, copy its files to a package, and generate a (basically static) package.py. The link you've sent doesn't try to copy files nor enclose the Python package and is more "live", which makes it fragile. rez-bind makes installed packages and the linked example looks like it's meant to be a source package.

Anyway, all I know is, if I want to show people Rez or adopt it at places, they'll want to build python and C++ code, usually. For gcc specifically, if the solution to deprecate rez-bind is "just build it yourself" - gcc takes 30-60 minutes to compile assuming it actually compiles without issue. That's going to throw a damper on anyone's enthusiasm to try new things. And if the process fails midway, even more so.

As a group, we need to make sure new users get the best possible experience, because that's typically who rez-bind is meant for.

[instinct-vfx]

Trying to adress the many points raised here. Please let me know if i miss things.

I think we still need to have a discussion as a community about what the alternative would look like. Personally, I've been having trouble convincing others on the merits of Rez because of certain factors which all contribute into a high barrier of entry. Unless the rez-bind alternative works 99.999% of the time, it'll frustrate new users more to remove rez-bind who are just looking to start / try stuff out.

I think this is the main source of confusion here. When we are talking about deprecating rez-bind that means the mechanism of providing a way to add pluggable, programatically created packages. Not the packages rez currently ships with (which are here: https://github.com/AcademySoftwareFoundation/rez/tree/master/src/rez/bind). We need to improve the docs, and we need to improve the off-the-shelf experience. I think everyone agrees on that.

would benefit from some updating since it's phrased as a non-optional step, far from being deprecated.

Totally agreed. There is a discussion topic to improve docs that has quite a bit of overlap with the points raised here already: #1277

If the version is not specified, it uses the latest released [...] no matter if it's resolvable or not.
That's expected behavior. No version specified always means "use latest" with rez. I am not sure about "resolvable or not". Could you explain what you mean there? If it is not resolvable it will not be found? Or do you mean a centrally released package that wraps a local install and that install not being available? That is not something the solver can know in advance which is one of the reasons we release full python packages with payloads. If you use python packages that wrap local installs then managing the local installs is out of scope for rez. Which can indeed be tricky (this sadly applies to all packages that can not be made portable and require local installation).

Since there's indeed no os/osx-12.4 in shared location, yet if there's no os package at all - it will fail to resolve somepackage.
That's the expected behavior? Can I use some "catch-all" version of os package since it does not care about version when implicit?

Implicit packages are simply packages that are added to every resolve if they are not part of the request anyways (they are added as weak requirements, see https://github.com/AcademySoftwareFoundation/rez/wiki/Basic-Concepts#implicit-packages). Implicit refers to them being implicitly added to your request, not the packages themselves. And as that makes them packages in your request they need to exist. rez does not care where as long as they exist in any of your configured packages_paths. The default config looks as follows:

implicit_packages = [
    "~platform=={system.platform}",
    "~arch=={system.arch}",
    "~os=={system.os}",
]

Which results in rez adding a request requesting your current platform/arch and os. And if these packages do not exist, it fails of course.

As a group, we need to make sure new users get the best possible experience, because that's typically who rez-bind is meant for.
I agree of course. Though rez-bind is just a way to create packages and not to be confused with the package examples it provides. So yes, i would agree that we need a better quickstart, but it does not matter if you provide a tool, a rez-bind setup or even readily built packages for all platforms. They are separate things.

Anyway, all I know is, if I want to show people Rez or adopt it at places, they'll want to build python and C++ code, usually. For gcc specifically, if the solution to deprecate rez-bind is "just build it yourself" - gcc takes 30-60 minutes to compile assuming it actually compiles without issue. That's going to throw a damper on anyone's enthusiasm to try new things. And if the process fails midway, even more so.

As mentioned above everything rez-bind does can be done with a tool using the API or standard packages. rez-bind could also build gcc from scratch if you adapted current implementation or replaced it with a custom one on plugin path. We need to provide sensible defaults of course and that has yet to be done, but the rez-bind ones are problematic in many ways. Running --quickstart on windows falls os its face unless being run with elevated privileges for example.

[instinct-vfx]

P.S. i removed the bug label because i don't think this can be considered a bug no matter how you look at it. Personally i also think this would be better suited for Github Discussions rather than an issue.

[JeanChristopheMorinPerso]

@instinct-vfx I converted it into a discussion as I also feel like it fits better into a discussion.

[instinct-vfx]

Thanks @JeanChristopheMorinPerso Had not seen that this is so easy. Noted for future use :)

[nerdvegas]

Hey all, My 2c: We deprecate rez-bind and instead we have a built-in repository of equivalent packages, such as python. I would keep this repo small (since package recipes really belong elsewhere - but in the interests of supporting a 'quickstart' with low barrier of entry, I think it's ok in this instance). These packages should be constrained to providing only simple things (os, platform), or things that bind to existing local installations (python), and this should be made clear. I know that this doesn't solve all problems, but I think that removing rez-bind would be a good first step, as it's so often a source of confusion. A

…

On Mon, Jun 20, 2022 at 9:01 AM Allan Johns ***@***.***> wrote: Hi Alexey, It's not the implicit-ness here that means `os` doesn't have to exist - it's the "weak reference" (~). See https://github.com/AcademySoftwareFoundation/rez/wiki/Basic-Concepts#weak-references . On Fri, Jun 17, 2022 at 2:42 PM Alexey Pelykh ***@***.***> wrote: > You will have to release all combinations of os/arch/platform to a shared > location. > > @instinct-vfx <https://github.com/instinct-vfx> but why if they are > implicit? rez-env somepackage --no-local gives > > requested packages: > somepackage > ~platform==osx (implicit) > ~arch==x86_64 (implicit) > ~os==osx-12.4 (implicit) > > but rez-env os --no-local > > 06:40:11 ERROR PackageNotFoundError: Package could not be found: os==osx-12.4 > > Since there's indeed no os/osx-12.4 in shared location, yet if there's > no os package at all - it will fail to resolve somepackage. > That's the expected behavior? Can I use some "catch-all" version of os > package since it does not care about version when implicit? > > — > Reply to this email directly, view it on GitHub > <#1332 (reply in thread)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AAMOUSSVO73K7423HU4XYKLVPP625ANCNFSM5ZAUZ77A> > . > You are receiving this because you were mentioned.Message ID: > <AcademySoftwareFoundation/rez/repo-discussions/1332/comments/2968732@ > github.com> >

[alexey-pelykh]

I've tried using the https://github.com/AcademySoftwareFoundation/rez/wiki/Package-Definition-Guide#example-package but I don't think the same is up to date - such package is not releasable to the shared repo and not resolvable by version, only rez-env python but not rez-env python-3.9.0-detected or similar. Not releasable because it will capture python version of the releasing machine. And what I'm after is something like "meta-package" that is resolving version every time it's resolved, not at build time

[alexey-pelykh]

But! If it would be a "combined" package as in

    Another supported storage format is to store all package versions within a
    single package family in one file, like so:

        /LOCATION/pkgC.yaml
        /LOCATION/pkgD.py

    These 'combined' package files allow for differences between package
    versions via a 'package_overrides' section:

        name: pkgC

        versions:
        - '1.0'
        - '1.1'
        - '1.2'

Then it would likely work, but then the referenced example is not about that

[nerdvegas]

Hi Alexey, It's not the implicit-ness here that means `os` doesn't have to exist - it's the "weak reference" (~). See https://github.com/AcademySoftwareFoundation/rez/wiki/Basic-Concepts#weak-references .

…

On Fri, Jun 17, 2022 at 2:42 PM Alexey Pelykh ***@***.***> wrote: You will have to release all combinations of os/arch/platform to a shared location. @instinct-vfx <https://github.com/instinct-vfx> but why if they are implicit? rez-env somepackage --no-local gives requested packages: somepackage ~platform==osx (implicit) ~arch==x86_64 (implicit) ~os==osx-12.4 (implicit) but rez-env os --no-local 06:40:11 ERROR PackageNotFoundError: Package could not be found: os==osx-12.4 Since there's indeed no os/osx-12.4 in shared location, yet if there's no os package at all - it will fail to resolve somepackage. That's the expected behavior? Can I use some "catch-all" version of os package since it does not care about version when implicit? — Reply to this email directly, view it on GitHub <#1332 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMOUSSVO73K7423HU4XYKLVPP625ANCNFSM5ZAUZ77A> . You are receiving this because you were mentioned.Message ID: <AcademySoftwareFoundation/rez/repo-discussions/1332/comments/2968732@ github.com>