Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

single multi-version package file for "external" packages #54

Closed
chadrik opened this issue Apr 29, 2014 · 2 comments
Closed

single multi-version package file for "external" packages #54

chadrik opened this issue Apr 29, 2014 · 2 comments

Comments

@chadrik
Copy link
Contributor

chadrik commented Apr 29, 2014

One feature that I'd really like to see in rez is the ability to manage external applications using a single multi-version package file.

A quick clarification of terms:

  • built package: a package that is built and installed to a central location using rez's build tools; typically contains all the resources for the packages (libraries, executables, includes) under the package's root directory.
  • external package: a package which is not built by rez, and whose resources typically exists outside of the central rez install path.

I've added this multi-version package feature in my own fork and used it for awhile, and it makes managing applications like maya and nuke much easier. Previously I had a directory structure like this:

maya/
├── 2012.17
│   │── package.yaml
│   │── platform-darwin
│   └── platform-linux
├── 2013.0
│   │── package.yaml
│   │── platform-darwin
│   └── platform-linux
├── 2014.0
│   │── package.yaml
│   └── platform-linux
├── 2014.2
│   │── package.yaml
│   └── platform-linux
├── 2014.4
│   │── package.yaml
│   └── platform-linux
├── 2014.50
│   │── package.yaml
│   └── platform-linux
├── 2014.51
│   │── package.yaml
│   └── platform-linux
└── 2014.59
    │── package.yaml
    └── platform-linux

(Note that all of those variant sub-directories are empty, they exist because rez errors if they don't. Also, I stopped using the ext symlink + '!ROOT!' because I found it hurt more than helped)

Now, with my fork, I have a single file called maya.yaml that looks like this:

name : maya

config_version : 0

versions :
- "2012.17"
- "2014.0"
- "2014.2"
- "2014.4"
- "2014.50"
- "2014.51"
- "2014.59"

# -- global settings:

variants:
- [ platform-linux ]

help: google-chrome http://download.autodesk.com/global/docs/maya2014/en_us/index.html

commands: |
  # TODO: support padding in rez
  env.MAYA_VERSION_NUM = version.major
  env.CMAKE_MODULE_PATH.append('{root}/cmake')

  if machine.platform == 'linux':
    env.MAYA_LOCATION = '/usr/autodesk/maya{version:#0.#01}-x64'
  elif machine.platform == 'darwin':
    env.MAYA_LOCATION = '/Applications/Autodesk/maya{version:#0.#01}/Maya.app/Contents'
  elif machine.platform == 'windows':
    env.MAYA_LOCATION = 'C:/Autodesk/maya{version.major}-x64'
  env.PATH.prepend('$MAYA_LOCATION/bin')
  env.MAYA_DEBUG_ENABLE_CRASH_REPORTING = 1

tools:
- maya
- mayapy

---

# -- version overrides:

version : 2012|2013

requires:
- python-2.6

# local variant override for 2012/2013:
# moving forward we will only support linux
variants:
- [ platform-linux ]
- [ platform-darwin ]

---

version : 2014

requires:
- python-2.7

The --- separates yaml sub-documents. The first sub-doc provides the global values that apply to all versions, and the remaining sub-docs provide version-specific overrides.

On the one hand, I know that package files are meant to be immutable once released, and while I try to stick to that rule for built packages, for external packages I find it very difficult. Adjustments sometimes need to be made that affect the package.yaml of every version of a package, or, even worse in terms of automation, most of them but not all. This single file gives a comprehensible birds-eye-view of the differences between versions. I put these external package definitions under version control and manage new releases and modifications by editing and committing these files.

Another perk is that when editing many rez package files it is much easier to keep track of what package you're editing, since they're not all named package.yaml.

One issue that I still need to address is time-stamping, as I'm still a novice with rez's timestamp mechanics. I'm hoping that these can be handled either automatically in an external side-car file (e.g. .maya-release.yaml) or by manually, by editing a timestamps metadata field in the package file. I'm open to suggestions.

To me this is a killer feature and I'd like for it to be included in rez, but I'd like to hear what people think.

@nerdvegas
Copy link
Contributor

Timestamping is pretty straightforward. Every package has (should have) an
epoch time associated with it, signifying when it came into existence. This
timestamp is stored in its own file in the rez package install (although I
think I'll change this so it goes into the new release-info.yaml file).
This isn't based on directory creation date or anything like that, because
more precise control is needed - the timestamp is set to some time just
after the package was completely installed, so that other packages can't
see it until it's ready to be consumed.

When you use timestamping in Rez, you are just ignoring all packages with a
timestamp later than the one you provide. This effectively rolls back time,
so you can reproduce the same resolve you got at some time in the past.
Packages must always support this - even though packages can have a
missing timestamp, this is only for backwards compatibility reasons, and
rez-2.0 currently defaults to printing warning messages when an
untimestamped package is found (although this is configurable).

Anyway onto multi-version package definitions:

They aren't my cup of tea personally, but as long as they were supported
within resources.py specifically (ie none of the rest of the code is aware
of the single yaml file), and as long as they supported timestamping, then
I don't see a problem with this. Although, this would only be supported in
yaml files, when I'd expect the majority of future rez packages to be
package.py-based. I don't know, personally I think it breaks the immutable
paradigm somewhat, and for how much gain really?

On Mon, Apr 28, 2014 at 6:35 PM, Chad Dombrova notifications@github.comwrote:

One feature that I'd really like to see in rez is the ability to manage
external applications using a single multi-version package file.

A quick clarification of terms:

  • built package: a package that is built and installed to a central
    location using rez's build tools; typically contains all the resources for
    the packages (libraries, executables, includes) under the package's root
    directory.
  • external package: a package which is not built by rez, and whose
    resources typically exists outside of the central rez install path.

I've added this multi-version package feature in my own fork and used it
for awhile, and it makes managing applications like maya and nuke much
easier. Previously I had a directory structure like this:

maya/
├── 2012.17
│ │── package.yaml
│ │── platform-darwin
│ └── platform-linux
├── 2013.0
│ │── package.yaml
│ │── platform-darwin
│ └── platform-linux
├── 2014.0
│ │── package.yaml
│ └── platform-linux
├── 2014.2
│ │── package.yaml
│ └── platform-linux
├── 2014.4
│ │── package.yaml
│ └── platform-linux
├── 2014.50
│ │── package.yaml
│ └── platform-linux
├── 2014.51
│ │── package.yaml
│ └── platform-linux
└── 2014.59
│── package.yaml
└── platform-linux

(Note that all of those variant sub-directories are empty, they exist
because rez errors if they don't. Also, I stopped using the ext symlink +
'!ROOT!' because I found it hurt more than helped)

Now, with my fork, I have a single file called maya.yaml that looks like
this:

name : maya
config_version : 0
versions :- "2012.17"- "2014.0"- "2014.2"- "2014.4"- "2014.50"- "2014.51"- "2014.59"

-- global settings:

variants:- [ platform-linux ]
help: google-chrome http://download.autodesk.com/global/docs/maya2014/en_us/index.html
commands: |

TODO: support padding in rez

env.MAYA_VERSION_NUM = version.major
env.CMAKE_MODULE_PATH.append('{root}/cmake')

if machine.platform == 'linux':
env.MAYA_LOCATION = '/usr/autodesk/maya{version:#0.#1}-x64'
elif machine.platform == 'darwin':
env.MAYA_LOCATION = '/Applications/Autodesk/maya{version:#0.#1}/Maya.app/Contents'
elif machine.platform == 'windows':
env.MAYA_LOCATION = 'C:/Autodesk/maya{version.major}-x64'
env.PATH.prepend('$MAYA_LOCATION/bin')
env.MAYA_DEBUG_ENABLE_CRASH_REPORTING = 1

tools:- maya- mayapy

-- version overrides:

version : 2012|2013
requires:- python-2.6

local variant override for 2012/2013:# moving forward we will only support linuxvariants:- [ platform-linux ]- [ platform-darwin ]


version : 2014
requires:- python-2.7

The --- separates yaml sub-documents. The first sub-doc provides the
global values that apply to all versions, and the remaining sub-docs
provide version-specific overrides.

On the one hand, I know that package files are meant to be immutable once
released, and while I try to stick to that rule for built packages, for
external packages I find it very difficult. Adjustments sometimes need to
be made that affect the package.yaml of every version of a package, or,
even worse in terms of automation, most of them but not all. This single
file gives a comprehensible birds-eye-view of the differences between
versions. I put these external package definitions under version control
and manage new releases and modifications by editing and committing these
files.

Another perk is that when editing many rez package files it is much easier
to keep track of what package you're editing, since they're not all named
package.yaml.

One issue that I still need to address is time-stamping, as I'm still a
novice with rez's timestamp mechanics. I'm hoping that these can be handled
either automatically in an external side-car file (e.g. .maya-release.yaml)
or by manually, by editing a timestamps metadata field in the package
file. I'm open to suggestions.

To me this is a killer feature and I'd like for it to be included in rez,
but I'd like to hear what people think.


Reply to this email directly or view it on GitHubhttps://github.com//issues/54
.

@chadrik
Copy link
Contributor Author

chadrik commented May 7, 2014

as long as they were supported within resources.py specifically (ie none of the rest of the code is aware of the single yaml file), and as long as they supported timestamping, then I don't see a problem with this.

should be doable. this will take a bit more work in my schema branch to represent hierarchies of files and folders as a single schema (see issue #52). in this way a single file, like the one described here, can provide the same metadata as a tree of files.

Although, this would only be supported in yaml files, when I'd expect the majority of future rez packages to be package.py-based.

I've created a python prototype that uses a context manager to track the overrides:

name = 'maya'

config_version = 0

versions = ["2012.17",
            "2014.2",
            "2014.4",
            "2014.50",
            "2014.51",
            "2014.59"]

variants = [ 'platform-linux' ]

help = 'google-chrome http://download.autodesk.com/global/docs/maya2014/en_us/index.html'

def commands():
    env.MAYA_VERSION_NUM = version.major
    env.CMAKE_MODULE_PATH.append('{root}/cmake')

    if machine.platform == 'linux':
        env.MAYA_LOCATION = '/usr/autodesk/maya{version:#0.#01}-x64'
    elif machine.platform == 'darwin':
        env.MAYA_LOCATION = '/Applications/Autodesk/maya{version:#0.#01}/Maya.app/Contents'
    elif machine.platform == 'windows':
        env.MAYA_LOCATION = 'C:/Autodesk/maya{version.major}-x64'
    env.PATH.prepend('$MAYA_LOCATION/bin')
    env.MAYA_DEBUG_ENABLE_CRASH_REPORTING = 1

tools = ['maya',
         'mayapy']

with overrides():

    version = '2012|2013'

    requires = ['python-2.6']

    # local override for 2012/2013:
    # moving forward we will only support linux
    variants = [
        [ 'platform-linux' ],
        [ 'platform-darwin' ]]

with overrides():

    version = '2014'

    requires = ['python-2.7']

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants