pex -c incorrectly alters exit codes

[rouge8]

Running pytest from a pex always exits 0 if there are test failures, but not if there are command line argument processing errors.

For example, given this test_fail.py:

assert False

A pytest pex will exit zero:

$ pex --version && pex pytest -c py.test -o pytest.pex && ./pytest.pex test_fail.py ; echo $?
pex 1.0.1
=========================================================================== test session starts ============================================================================
platform darwin -- Python 2.7.10 -- py-1.4.30 -- pytest-2.7.2
rootdir: /Users/rouge8/forks/pex, inifile:
collected 0 items / 1 errors

================================================================================== ERRORS ==================================================================================
______________________________________________________________________ ERROR collecting test_fail.py _______________________________________________________________________
test_fail.py:1: in <module>
    assert False
E   assert False
========================================================================= 1 error in 0.00 seconds ==========================================================================
0

But running py.test in a virtualenv or similar will exit 1.

On the other hand, giving a bad option to py.test will exit non-zero, as expected:

$ pex --version && pex pytest -c py.test -o pytest.pex && ./pytest.pex --bad-option test_fail.py ; echo $?
pex 1.0.1
usage: pytest.pex [options] [file_or_dir] [file_or_dir] [...]
pytest.pex: error: unrecognized arguments: --bad-option
2

[wickman]

I'm able to repro. This is probably a bug with "-c". If you do "pex pytest -m pytest -o pytest.pex" then it works:

mba=2015.07.29.pex137=; pex pytest -m pytest -o pytest.pex
mba=2015.07.29.pex137=; ./pytest.pex test_fail.py
====================================================== test session starts =======================================================
platform darwin -- Python 2.7.9 -- py-1.4.30 -- pytest-2.7.2
rootdir: /Users/wickman/work/2015.07.29.pex137, inifile: 
collected 1 items 

test_fail.py F

============================================================ FAILURES ============================================================
___________________________________________________________ test_fail ____________________________________________________________

    def test_fail():
>     assert False
E     assert False

test_fail.py:2: AssertionError
==================================================== 1 failed in 0.01 seconds ====================================================
mba=2015.07.29.pex137=; echo $?
1

[rouge8]

Yup, using -m pytest instead of -c py.test works. Thanks!

[rouge8]

I looked into this a bit this morning:

• pex -c py.test finds the console script pytest:main, which returns an integer exit code
• pex -m pytest hits pytest.py which raises SystemExit with that integer exit code

According to the setuptools docs:

The functions you specify are called with no arguments, and their return value is passed to sys.exit(), so you can return an errorlevel or message to print to stderr

pex currently just calls the entrypoint function but does nothing with the return value. I would guess returning rather than exiting from the entrypoint is uncommon (pex.bin.pex:main calls sys.exit() itself), which is why I only saw this with py.test.

I'm not sure if I'll have time to work on a fix immediately, but hopefully this writeup is helpful for anyone who does.