devenv test reporting success even when errors and failures occur

[kourtni]

Describe the bug
When running devenv test, failures and errors can get swallowed resulting in devenv test reporting that "Tests passed".

To reproduce
This gist, https://gist.github.com/kourtni/7ee2749b0bb75253e0d29d2deab1b4fb, is a complete example that reproduces the issue. I expect that running devenv test from that environment should fail, however it currently passes.

Version
devenv 1.0.5 (x86_64-linux)

[domenkozar]

Could you also attach the python files so I can reproduce the failure?

[kourtni]

Hi,

The issue is that devenv test swallows errors and test failures if enterTest ends with a command that exits normally.

The screenshot is from tests run in a private repo, so I can't share those files. But as I indicated in the Gist, https://gist.github.com/kourtni/7ee2749b0bb75253e0d29d2deab1b4fb, this behavior is completely reproducible without actually requiring any python files. In fact the point of that Gist is to show that devenv test passes even without ANY of the required packages installed or source files available. I am able to reproduce the issue with only the devenv.nix and devenv.yaml files included in the Gist, and then simply running devenv test.

But if you absolutely want to include a dummy test for some reason, you could create a file with a simple test where you assert 1 == 2 or something that will obviously fail.

As long as the final command in enterTest is an echo or some other command guaranteed to succeed, then any failures that occurred prior will be swallowed and the devenv test will complete in a Tests passed state.

[sandydoo]

@kourtni, the issue isn't the echo, it's the &&.

We set the shell to exit early on errors with set -e, but this flag does not apply in a few situations. When using &&, only the last command affects the failure of the script. When the first poetry command fails, set -e is ignored and the second command is never run because the first failed. One solution is to not use && and place the second poetry command on the next line.

Bash is not a great environment for testing because of this. Perhaps there are testing tools out there to fix this.

[kourtni]

OK, thanks for those details @sandydoo!

Perhaps it makes sense to add that caveat to the devenv test documentation, so that people aren't scratching their heads like I was.

[joeapearson]

I find a similar issue; pre-commit hooks (run with devenv test) exit with code 0 even if the hooks themselves exit with failure.

The issue seems related to the one originally reported hence commenting here. Here's a gist to demonstrate, which includes the devenv test output that I get.

Is this expected? If yes, perhaps the documentation should be updated?

devenv version
devenv 1.3.0 (aarch64-darwin)

[domenkozar]

Fixed in #1510