rez-build breaks if "|" in a required package's version on Windows

[ColinKennedy]

The | "or" feature in https://github.com/nerdvegas/rez/wiki/Basic-Concepts#package-requests does not work on Windows 10, Rez 2.26.1. Specifically, this syntax resolves a context correctly but raises an error whenever I try to rez-build a package:

requires = [
    'package_a==|>=0.1',
]

If I have two rez packages, package_a and package_b

And their files look like this:

package_a/package.py

name = 'package_a'

version = '0.1.0'

build_command = 'python {root}/rezbuild.py {install}'


def commands():
    pass

package_b/package.py

name = 'package_b'

version = '0.1.0'

build_command = 'python {root}/rezbuild.py {install}'

requires = [
    'package_a==|>=0.1',
]

def commands():
    pass

And both packages have a rezbuild.py file that looks like this:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

# IMPORT STANDARD LIBRARIES
import sys
import os

def build(source_path, build_path, install_path, targets):
    pass


if __name__ == '__main__':
    build(
        source_path=os.environ['REZ_BUILD_SOURCE_PATH'],
        build_path=os.environ['REZ_BUILD_PATH'],
        install_path=os.environ['REZ_BUILD_INSTALL_PATH'],
        targets=sys.argv[1:],
    )

If I run rez-build while cd'ed into package_b on Linux, it builds correctly with no issue. This same setup fails on Windows with the error '0.1+' is not recognized as an internal or external command, operable program or batch file. I've tried a few variations and they all seem to have the same issue. The package is resolved correctly but breaks once it tries to run the build system. It looks like rez-build is interpreting the | as a separator for a command and 0.1 as a literal Windows command.

Here is a .zip of the two Rez packages which can be used to reproduce the error and the full output log of the rez-build command I ran:

rez_release_test.zip
rez_build_windows_command_error.txt

[ColinKennedy]

Updated once I realized that the issue is not with rez-release but with rez-build.

[nerdvegas]

Could you provide a stacktrace? You may need to run with REZ_DEBUg envvar set to 1. Cheers A

…

On Tue, Jan 15, 2019 at 5:38 AM Colin Kennedy ***@***.***> wrote: An example syntax in https://github.com/nerdvegas/rez/wiki/Basic-Concepts#package-requests does not work on Windows. If I have two rez packages, package_a and package_b And their files look like this: package_a/package.py name = 'package_a' version = '0.1.0' build_command = 'python {root}/rezbuild.py {install}' def commands(): pass package_b/package.py name = 'package_b' version = '0.1.0' build_command = 'python {root}/rezbuild.py {install}' requires = [ 'package_a==|>=0.1', ] def commands(): pass And both packages have a rezbuild.py file that looks like this: #!/usr/bin/env python # -*- coding: utf-8 -*- # IMPORT STANDARD LIBRARIES import sys import os def build(source_path, build_path, install_path, targets): pass if __name__ == '__main__': build( source_path=os.environ['REZ_BUILD_SOURCE_PATH'], build_path=os.environ['REZ_BUILD_PATH'], install_path=os.environ['REZ_BUILD_INSTALL_PATH'], targets=sys.argv[1:], ) If I run rez-release on Linux, it releases correctly with no issue. This same setup fails on Windows with the error '0.1+' is not recognized as an internal or external command, operable program or batch file. I've tried a few variation syntaxes and they all seem to have the same issue. It looks like rez-release is interpreting the | as a separator for a command and 0.1 as a literal Windows command. Here is a .zip of an example project to reproduce it and the full output log of the rez-release command I ran: rez_release_test.zip <https://github.com/nerdvegas/rez/files/2756575/rez_release_test.zip> rez_release_windows_command_error.txt <https://github.com/nerdvegas/rez/files/2756589/rez_release_windows_command_error.txt> — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#558>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABjqSs02QVXwbcX2nUltPQjSrMdiZwOdks5vDM6sgaJpZM4Z_aab> .

[ColinKennedy]

Sure thing, here's the same command from before with REZ_DEBUG=1

rez_build_windows_command_error_with_stack_trace.txt

Thanks Allan

[nerdvegas]

Ok, so I have an inkling that this isn't related to build, but is instead an escape issue in the cmd code generated by the context. Sorry I can't debug this directly, I don't develop on Windows. Could you try doing a rez-env with an ORd request like this? Eg 'rez-env "foo-1|2"'. I'm assuming you'll see the same error. If you do, could you then try: ]$ rez-env 'foo-1|2' -o test.rxt ]$ rez-context -i test.rxt That will print out the cmd code that is sourced to create the resolved env, and it should become obvious at that point if the '|' isn't getting correctly escaped. I think this is what's happening, because it'd explain why '0.1+' is getting interpreted as a command. Thx A

…

On Tue, Jan 15, 2019 at 5:38 AM Colin Kennedy ***@***.***> wrote: An example syntax in https://github.com/nerdvegas/rez/wiki/Basic-Concepts#package-requests does not work on Windows. If I have two rez packages, package_a and package_b And their files look like this: package_a/package.py name = 'package_a' version = '0.1.0' build_command = 'python {root}/rezbuild.py {install}' def commands(): pass package_b/package.py name = 'package_b' version = '0.1.0' build_command = 'python {root}/rezbuild.py {install}' requires = [ 'package_a==|>=0.1', ] def commands(): pass And both packages have a rezbuild.py file that looks like this: #!/usr/bin/env python # -*- coding: utf-8 -*- # IMPORT STANDARD LIBRARIES import sys import os def build(source_path, build_path, install_path, targets): pass if __name__ == '__main__': build( source_path=os.environ['REZ_BUILD_SOURCE_PATH'], build_path=os.environ['REZ_BUILD_PATH'], install_path=os.environ['REZ_BUILD_INSTALL_PATH'], targets=sys.argv[1:], ) If I run rez-release on Linux, it releases correctly with no issue. This same setup fails on Windows with the error '0.1+' is not recognized as an internal or external command, operable program or batch file. I've tried a few variation syntaxes and they all seem to have the same issue. It looks like rez-release is interpreting the | as a separator for a command and 0.1 as a literal Windows command. Here is a .zip of an example project to reproduce it and the full output log of the rez-release command I ran: rez_release_test.zip <https://github.com/nerdvegas/rez/files/2756575/rez_release_test.zip> rez_release_windows_command_error.txt <https://github.com/nerdvegas/rez/files/2756589/rez_release_windows_command_error.txt> — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#558>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABjqSs02QVXwbcX2nUltPQjSrMdiZwOdks5vDM6sgaJpZM4Z_aab> .

[ColinKennedy]

You're right, the issue is reproduceable with rez-env

Here's the full output of the example you mentioned:

C:\Users\selecaotwo\temp\rez_release_test\package_b>rez-env "package_a==|>=0.1" -o test.rxt

C:\Users\selecaotwo\temp\rez_release_test\package_b>rez-context -i test.rxt
set REZ_RXT_FILE=test.rxt
REM
REM
REM ################################################################################
REM system setup
REM ################################################################################
set REZ_USED=c:\users\selecaotwo\rez\lib\site-packages\rez-2.26.1-py2.7.egg\rez
set REZ_USED_VERSION=2.26.1
set REZ_USED_TIMESTAMP=1547509186
set REZ_USED_REQUESTED_TIMESTAMP=0
set REZ_USED_REQUEST=package_a==|0.1+
set REZ_USED_IMPLICIT_PACKAGES=~platform==windows ~arch==AMD64 ~os==windows-10.0.17134
set REZ_USED_RESOLVE=package_a-0.1.0
set REZ_USED_PACKAGES_PATH=C:\Users\korinkite\temp\rez_release_test;C:\Users\selecaotwo\.rez\packages\int
set REZ_VERSION=2.26.1
set REZ_PATH=c:\users\selecaotwo\rez\lib\site-packages\rez-2.26.1-py2.7.egg\rez
set REZ_REQUEST=package_a==|0.1+ ~platform==windows ~arch==AMD64 ~os==windows-10.0.17134
set REZ_RESOLVE=package_a-0.1.0
set REZ_RAW_REQUEST=package_a==|0.1+ ~platform==windows ~arch==AMD64 ~os==windows-10.0.17134
set REZ_RESOLVE_MODE=latest
REM
REM
REM ################################################################################
REM package variables
REM ################################################################################
REM
REM variables for package package_a-0.1.0[]
REM --------------------------------------------------------------------------------
set REZ_PACKAGE_A_VERSION=0.1.0
set REZ_PACKAGE_A_MAJOR_VERSION=0
set REZ_PACKAGE_A_MINOR_VERSION=1
set REZ_PACKAGE_A_PATCH_VERSION=0
set REZ_PACKAGE_A_BASE=C:\Users\selecaotwo\.rez\packages\int\package_a\0.1.0
set REZ_PACKAGE_A_ROOT=C:\Users\selecaotwo\.rez\packages\int\package_a\0.1.0
REM
REM
REM ################################################################################
REM commands
REM ################################################################################
REM
REM commands from package package_a-0.1.0[]
REM --------------------------------------------------------------------------------
REM
REM
REM ################################################################################
REM post system setup
REM ################################################################################
set PATH=C:\Users\selecaotwo\AppData\Local\Pandoc\;C:\Program Files (x86)\GnuWin32\bin;C:\Program Files (x86)\Vim\vim81;_C:\Users\selecaotwo\rez-python_2.7.0_64_bit\Scripts\rez;%USERPROFILE%\AppData\Local\Microsoft\WindowsApps;C:\WINDOWS\System32\OpenSSH\;C:\Users\selecaotwo\AppData\Local\Programs\Python\Launcher\;C:\WINDOWS;C:\Program Files\git\mingw64\bin;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\system32;C:\Python27;C:\Program Files\Git\cmd;C:\Program Files\Git\mingw64\bin;C:\WINDOWS\System32\Wbem;C:\Users\selecaotwo\env\config\setpkg\bin;C:\Python27\Scripts;C:\Program Files\Git\usr\bin;C:\Users\selecaotwo\AppData\Local\Programs\MiKTeX 2.9\miktex\bin\x64\;C:\Program Files\git\ming64\bin;C:\Users\selecaotwo\rez\Scripts\rez;C:\Program Files\ag\bin;C:\Program Files\CMake\bin;C:\Program Files\fzf\bin
set PATH=%PATH%;c:\users\selecaotwo\rez\scripts\rez


C:\Users\selecaotwo\temp\rez_release_test\package_b>

Please let me know if you need any more information

[nerdvegas]

Yeah it looks fairly obvious that the string escaping implementation for cmd shell is to blame. That REZ_REQUEST set should be double quoted. If you look at rezplugins/shell/cmd.py's function 'escape_string', and compare it to sh.py's, you'll see the issue. The cmd shell doesn't appear to take as much consideration, and doesn't differentiate between literal and expandable strings. This needs to be fixed. Hth A

…

On Tue, Jan 15, 2019 at 11:00 AM Colin Kennedy ***@***.***> wrote: You're right, the issue is reproduceable with rez-env Here's the full output of the example you mentioned: C:\Users\selecaotwo\temp\rez_release_test\package_b>rez-env "package_a==|>=0.1" -o test.rxt C:\Users\selecaotwo\temp\rez_release_test\package_b>rez-context -i test.rxt set REZ_RXT_FILE=test.rxt REM REM REM ################################################################################ REM system setup REM ################################################################################ set REZ_USED=c:\users\selecaotwo\rez\lib\site-packages\rez-2.26.1-py2.7.egg\rez set REZ_USED_VERSION=2.26.1 set REZ_USED_TIMESTAMP=1547509186 set REZ_USED_REQUESTED_TIMESTAMP=0 set REZ_USED_REQUEST=package_a==|0.1+ set REZ_USED_IMPLICIT_PACKAGES=~platform==windows ~arch==AMD64 ~os==windows-10.0.17134 set REZ_USED_RESOLVE=package_a-0.1.0 set REZ_USED_PACKAGES_PATH=C:\Users\korinkite\temp\rez_release_test;C:\Users\selecaotwo\.rez\packages\int set REZ_VERSION=2.26.1 set REZ_PATH=c:\users\selecaotwo\rez\lib\site-packages\rez-2.26.1-py2.7.egg\rez set REZ_REQUEST=package_a==|0.1+ ~platform==windows ~arch==AMD64 ~os==windows-10.0.17134 set REZ_RESOLVE=package_a-0.1.0 set REZ_RAW_REQUEST=package_a==|0.1+ ~platform==windows ~arch==AMD64 ~os==windows-10.0.17134 set REZ_RESOLVE_MODE=latest REM REM REM ################################################################################ REM package variables REM ################################################################################ REM REM variables for package package_a-0.1.0[] REM -------------------------------------------------------------------------------- set REZ_PACKAGE_A_VERSION=0.1.0 set REZ_PACKAGE_A_MAJOR_VERSION=0 set REZ_PACKAGE_A_MINOR_VERSION=1 set REZ_PACKAGE_A_PATCH_VERSION=0 set REZ_PACKAGE_A_BASE=C:\Users\selecaotwo\.rez\packages\int\package_a\0.1.0 set REZ_PACKAGE_A_ROOT=C:\Users\selecaotwo\.rez\packages\int\package_a\0.1.0 REM REM REM ################################################################################ REM commands REM ################################################################################ REM REM commands from package package_a-0.1.0[] REM -------------------------------------------------------------------------------- REM REM REM ################################################################################ REM post system setup REM ################################################################################ set PATH=C:\Users\selecaotwo\AppData\Local\Pandoc\;C:\Program Files (x86)\GnuWin32\bin;C:\Program Files (x86)\Vim\vim81;_C:\Users\selecaotwo\rez-python_2.7.0_64_bit\Scripts\rez;%USERPROFILE%\AppData\Local\Microsoft\WindowsApps;C:\WINDOWS\System32\OpenSSH\;C:\Users\selecaotwo\AppData\Local\Programs\Python\Launcher\;C:\WINDOWS;C:\Program Files\git\mingw64\bin;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\system32;C:\Python27;C:\Program Files\Git\cmd;C:\Program Files\Git\mingw64\bin;C:\WINDOWS\System32\Wbem;C:\Users\selecaotwo\env\config\setpkg\bin;C:\Python27\Scripts;C:\Program Files\Git\usr\bin;C:\Users\selecaotwo\AppData\Local\Programs\MiKTeX 2.9\miktex\bin\x64\;C:\Program Files\git\ming64\bin;C:\Users\selecaotwo\rez\Scripts\rez;C:\Program Files\ag\bin;C:\Program Files\CMake\bin;C:\Program Files\fzf\bin set PATH=%PATH%;c:\users\selecaotwo\rez\scripts\rez C:\Users\selecaotwo\temp\rez_release_test\package_b> Please let me know if you need any more information — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#558 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABjqSiNmuoZPuiWxhmrZIpkebDkD4DPmks5vDRosgaJpZM4Z_aab> .

[ColinKennedy]

I see now. I changed escape_string so that it accounts for | with double quotes and I am now able to use rez-env and rez-release again as normal.

Something that is interesting though is that my solution still doesn't let me rez-build --install. If I do, I get a "Internal failure - expected installed variant" error. It may be an issue that's specific to my package though.

I'd be excited to get a patch of rez that includes a fix for this but this stopgap will work for us in the meantime. Thanks again for your help!

[nerdvegas]

Re your error: "Internal failure - expected installed variant". Please be aware that a recent version ( https://github.com/nerdvegas/rez/releases/tag/2.24.0) introduced code that affects build/release, and variant installation in general. If you are still getting this problem, could you try out the following release and see if it works: https://github.com/nerdvegas/rez/releases/tag/2.23.1. You may have found an edge case (the tests pass however). RE the windows string formatting issue, I've created a ticket: #564 Thx A

…

On Thu, Jan 17, 2019 at 6:45 AM Colin Kennedy ***@***.***> wrote: I see now. I changed escape_string so that it accounts for | with double quotes and I am now able to use rez-env and rez-release again. Something that is interesting though is that my solution still doesn't let me rez-build --install. If I do, I get a "Internal failure - expected installed variant" error. It may be an issue that's specific to my package though. I'd be excited to get a patch of rez that includes a fix for this but this stopgap will work for us in the meantime. Thanks again for your help! — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#558 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABjqSqXDnS8lW8Dev149kOwMCdhUYHskks5vD4FigaJpZM4Z_aab> .

[mottosso]

I changed escape_string so that it accounts for | with double quotes and I am now able to use rez-env and rez-release again as normal.

Just ran into this myself, what did you change it to @ColinKennedy?

Here's the current version, my regular expression-fu is too weak.

https://github.com/nerdvegas/rez/blob/3a5afd86dbf88801a44d2fc4f828140ef1be6b34/src/rezplugins/shell/cmd.py#L27-L28

[ColinKennedy]

Hey @mottosso

My solution was hacky so I don't recommend it for production code. It was a temp fix. That said, I haven't needed to use Rez on Windows for a while so this may not still work.

ColinKennedy@f0e0556

[mottosso]

Thanks @ColinKennedy, that does work, but yes it's unfortunate it's a second parsing step rather than part of the regular expression itself. I managed to find a regular expression for it and made a PR here: #558