Exit code test failure on OSX

[jakepruitt]

Seeing three failing tests on OSX 10.11.6:

Failures:

1) Scenario: osrm-contract - Non-existing option - features/options/contract/invalid.feature:7
   Step: And it should exit with code 1 - features/options/contract/invalid.feature:12
   Step Definition: features/step_definitions/options.js:29
   Message:
     AssertionError: 134 == 1
         at World.<anonymous> (/Users/jake/Projects/osrm-backend/features/step_definitions/options.js:30:16)
         at nextTickCallbackWith0Args (node.js:420:9)
         at process._tickCallback (node.js:349:13)

2) Scenario: osrm-extract - Non-existing option - features/options/extract/invalid.feature:7
   Step: And it should exit with code 1 - features/options/extract/invalid.feature:12
   Step Definition: features/step_definitions/options.js:29
   Message:
     AssertionError: 134 == 1
         at World.<anonymous> (/Users/jake/Projects/osrm-backend/features/step_definitions/options.js:30:16)
         at nextTickCallbackWith0Args (node.js:420:9)
         at process._tickCallback (node.js:349:13)

3) Scenario: osrm-routed - Non-existing option - features/options/routed/invalid.feature:7
   Step: And it should exit with code 1 - features/options/routed/invalid.feature:12
   Step Definition: features/step_definitions/options.js:29
   Message:
     AssertionError: 134 == 1
         at World.<anonymous> (/Users/jake/Projects/osrm-backend/features/step_definitions/options.js:30:16)
         at nextTickCallbackWith0Args (node.js:420:9)
         at process._tickCallback (node.js:349:13)

712 scenarios (3 failed, 709 passed)
3304 steps (3 failed, 3301 passed)

Looks like the exit code of bad commands is 134 rather than 1. When running the command in a stand-alone way I get:

libc++abi.dylib: terminating with uncaught exception of type boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::program_options::unknown_option> >: unrecognised option '--fly-me-to-the-moon'
Abort trap: 6

cc/ @danpat @springmeyer

[oxidase]

@jakepruitt exit code 134 is an exit due to SIGABRT (n = 6 ) http://tldp.org/LDP/abs/html/exitcodes.html

128+n Fatal error signal "n"

[danpat]

These failures are due to this change: 0b86896 where we're not catching std::exception any more. This triggers abort() when an exception is thrown, and the return code is platform specific (SIGABRT doesn't return 134 on OSX :-( ).

This is somewhat intended behaviour, as we'd like OSRM to exit noisily for all but a few exceptions. Exiting with SIGABRT triggers a core dump, and from that, we can easily script GDB to get stack traces. Our production systems automate this. Previously, when we caught std::exception, it masked the root cause of several problems.

@jakepruitt The test cases should handle this particular error - the tests that are failing here are when invalid parameters are passed on the command line.

https://github.com/Project-OSRM/osrm-backend/blob/master/features/step_definitions/options.js#L33-L35

https://github.com/Project-OSRM/osrm-backend/blob/master/features/options/extract/invalid.feature#L12

I thought I fixed this with this change: 176c224#diff-2d3b7ab3f92e44dc1a5f49fdefc0336fR128

but it looks like I didn't handle all cases. The SIGABRT exit code is platform specific, so these tests sometimes pass, depending on where you're running them. I'm surprised this slipped by Travis.

What we really should be doing here is catching boost::program_options::error and throwing a known error code (probably 1). This would make the tests more consistent for these "invalid parameter" cases, and still leave us with the "abort() on other exceptions" behaviour.

Ideally we'd have much more comprehensive error handling, but that's a bigger job....

[TheMarex]

In #2795 I had to work around this by defining a more general test function "it exists with an error" instead of "it exists with code \d".

I defined as an error condition as code != 0 || signal != '' (node will on my system set the exit code to 0 even if its SIGABRT)