hpcrun should identify when hpcfnbounds fails to start

[jmellorcrummey]

Today at LLNL, we encountered a problem where hpcfnbounds linked in a copy of libdw that was built with spack. The libdw copy was safely installed in lib/hpctoolkit/ext-libs. However, spack had hardcoded a dependence on a version of libbz2.so.1.0 that is in the spack install tree.

As a consequence of LLNL's default security, that spack repository, which was built in /usr/workspace/.../${LOGNAME} wasn't readable by others. The version of libbzip that was linked into libdw had incompatible version numbers from the ones publicly available in /lib64/libbzip.so.1 and /lib64/libbzip.so.1.0.6 and . As a result, hpcfnbounds-bin silently failed to run because it couldn't load a dependent library. hpcrun, however, continued to spawn failing copies of hpcfnbounds until the whole execution keeled over with an error message about something else, which wasn't helpful for diagnosing the problem.

hpcrun should detect that hpcfnbounds starts correctly - a handshake before using it for the first time would do this - and then, if something is amiss, rather than failing inscrutably after causing a fork bomb, it should exit with a helpful error message

hpcrun: unable to start hpcfnbounds to analyze application load modules
     check permissions of the script, the executable in libexec/hpctoolkit/hpcfnbounds-bin, 
     and its dependent libraries

[mwkrentel]

First, it's subtle and not really documented, but I suspect that your
toolkit build is misconfigured. What does the summary in config.log
say about the prereq paths?

You need to add --with-bzip=PATH to configure. Plain externals
doesn't need this because externals doesn't build a separate bzip, so
it just gets the system version.

But if you're building libdw from spack, especialy from my recipes,
then you need --with-bzip. That will tell toolkit to copy libbz2 to
ext-libs where it should find the right version.

Actually, could you repeat your build with --with-bzip and let me know
if that works? I'd like to test if all the parts for libdw and bzip
are in place. It's supposed to work, but I don't have a good way to
test it because it works for me even when misconfigured. (I don't
have a compute node with directories that aren't mounted.)

But you're right that we should handle this kind of failure more
gracefully. I'll fix it.

[mwkrentel]

Fixed in commit 735327a.

1. If the fnbounds server crashes or we shut it down, then use
   waitpid() to reduce zombies.

2. If the server doesn't even start up, then exit the process, the
   profile would be useless.