Infrastructure for PRRTE CI Tests #49

jjhursey · 2019-12-20T04:08:55Z

Define a few CI tests for PRRTE
See README.md for how to drop in a new test.

prrte/hello_world/hello.c

prrte/cycle/run.sh

jjhursey · 2019-12-20T18:04:00Z

@rhc These are the tests I've started with. I have a bit more to do in this PR before it is ready to merge. I'll return to it once I'm back from break.

I did notice a couple of failures with the current PRRTE master that need some looking into that can be recreated with these test cases.

in the hello_world example I try to access PMIX_LOCAL_RANK from a process on a non-launcher node and PMIx is returning not found. I commented it out in the current state of the test, but I think we should add it back in once it is fixed in PRRTE.
The hello_world example will often hang in the CI environment. I'm not sure why at the moment. It's not 100% of the time though. Because of this I am not going to enable it for PRRTE yet - instead I'll leave it as is until I can investigate when I get back.
The cycle example usually fails after about 10 iterations with the prted crashing. I'm not sure what's happening there.

I'll pick up on this when I get back from break.

jjhursey · 2020-01-08T20:21:30Z

PR openpmix/prrte#296 fixes the PMIX_LOCAL_RANK. I'll re-push the hello world with this value in it.

jjhursey · 2020-01-09T18:44:53Z

Ci is passing now for both of these tests. I'll work on finishing up the test case infrastructure and add a few more tests to the mix. Then I think this is ready to go.

jjhursey · 2020-01-22T21:45:40Z

bot:ibm:retest

jjhursey · 2020-01-22T21:50:35Z

bot:ibm:retest

jjhursey · 2020-01-22T22:27:05Z

bot:ibm:retest

jjhursey · 2020-01-23T14:27:00Z

bot:ibm:retest

jjhursey · 2020-01-29T20:45:22Z

bot:ibm:retest

In the prior CI test the prte daemon crashed after emitting these warnings a few times:

--------------------- Execution (hostname): 16
[warn] Epoll ADD(4) on fd 34 failed. Old events were 0; read change was 0 (none); write change was 1 (add); close change was 0 (none): Bad file descriptor
[331e97292fa3:00130] PRRTE ERROR: Bad parameter in file base/iof_base_output.c at line 267

Then this stack trace:

--------------------- Execution (hostname): 84
[331e97292fa3:00130] *** Process received signal ***
[331e97292fa3:00130] Signal: Segmentation fault (11)
[331e97292fa3:00130] Signal code: Address not mapped (1)
[331e97292fa3:00130] Failing at address: 0x30
[331e97292fa3:00130] [ 0] [0x3fffb4890478]
[331e97292fa3:00130] [ 1] /workspace/exports/pmix/lib/pmix/mca_bfrops_v4.so(mca_bfrops_v4_component+0x108)[0x3fffb3d601c0]
[331e97292fa3:00130] [ 2] /workspace/exports/prrte/lib/libprrte.so.2(pmix_server_stdin_fn+0x70)[0x3fffb47930f0]
[331e97292fa3:00130] [ 3] /workspace/exports/pmix/lib/libpmix.so.0(+0xd11d4)[0x3fffb46011d4]
[331e97292fa3:00130] [ 4] 331e97292fa3
/workspace/exports/pmix/lib/libpmix.so.0(+0xb3244)[0x3fffb45e3244]
[331e97292fa3:00130] [ 5] /workspace/exports/pmix/lib/libpmix.so.0(+0xb3b60)[0x3fffb45e3b60]
[331e97292fa3:00130] [ 6] /workspace/exports/pmix/lib/libpmix.so.0(pmix_ptl_base_process_msg+0x3a4)[0x3fffb46b3a7c]
[331e97292fa3:00130] [ 7] /opt/hpc/local/libevent/lib/libevent-2.1.so.6(+0x2b918)[0x3fffb449b918]
[331e97292fa3:00130] [ 8] /opt/hpc/local/libevent/lib/libevent-2.1.so.6(event_base_loop+0x504)[0x3fffb449c404]
[331e97292fa3:00130] [ 9] /workspace/exports/pmix/lib/libpmix.so.0(+0xe5d50)[0x3fffb4615d50]
[331e97292fa3:00130] [10] /lib64/libpthread.so.0(+0x8cd4)[0x3fffb4278cd4]
[331e97292fa3:00130] [11] /lib64/libc.so.6(clone+0xe4)[0x3fffb41a7e94]
[331e97292fa3:00130] *** End of error message ***
[331e97292fa3:00645] PMIX ERROR: UNPACK-PAST-END in file common/pmix_iof.c at line 1116
./run.sh: line 43:   130 Segmentation fault      (core dumped) prte --hostfile $CI_HOSTFILE

jjhursey · 2020-01-29T20:59:43Z

Same failure in CI. I'll try to investigate why prte is falling over.

jjhursey · 2020-01-29T22:33:51Z

I got a core from the failure - backtrace below. CI used an optimized build of PRRTE, so the core is more challenging to use. I am not able to reproduce in a debug build. I suspect that the bad FD error that proceeds it is then causing the stdio write to access bad memory.

(gdb) thread apply all bt

Thread 4 (Thread 0x3fffadaaf1b0 (LWP 135)):
#0  0x00003fffaf3a9f28 in select () from /lib64/libc.so.6
#1  0x00003fffadab9a34 in listen_thread () from /workspace/exports/prrte/lib/prrte/mca_oob_tcp.so
#2  0x00003fffaf488cd4 in start_thread () from /lib64/libpthread.so.0
#3  0x00003fffaf3b7e94 in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x3fffae32f1b0 (LWP 134)):
#0  0x00003fffaf3a9f28 in select () from /lib64/libc.so.6
#1  0x00003fffaf8c536c in listen_thread (obj=0x0) at base/ptl_base_listener.c:214
#2  0x00003fffaf488cd4 in start_thread () from /lib64/libpthread.so.0
#3  0x00003fffaf3b7e94 in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x3fffafb069a0 (LWP 131)):
#0  0x00003fffaf48e92c in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
#1  0x00003fffafa56f0c in prrte_state_base_track_procs () from /workspace/exports/prrte/lib/libprrte.so.2
#2  0x00003fffaf6ab918 in event_process_active_single_queue (base=0x10023d01eb0, activeq=0x10023d02580, max_to_process=2147483647, endtime=0x0)
    at event.c:1646
#3  0x00003fffaf6ac404 in event_process_active (base=0x10023d01eb0) at event.c:1738
#4  event_base_loop (base=0x10023d01eb0, flags=<optimized out>) at event.c:1961
#5  0x0000000010002728 in main ()

Thread 1 (Thread 0x3fffaebff1b0 (LWP 133)):
#0  0x00003fffad0941ac in push_stdin () from /workspace/exports/prrte/lib/prrte/mca_iof_hnp.so
#1  0x00003fffaf9a30f0 in pmix_server_stdin_fn () from /workspace/exports/prrte/lib/libprrte.so.2
#2  0x00003fffaf8111d4 in pmix_server_iofstdin (peer=0x3fffa031e4f0, buf=0x3fffaebfe410, cbfunc=0x3fffaf7dfb34 <op_cbfunc>, 
    cbdata=0x3fffa007df10) at server/pmix_server_ops.c:3853
#3  0x00003fffaf7f3244 in server_switchyard (peer=0x3fffa031e4f0, tag=104, buf=0x3fffaebfe410) at server/pmix_server.c:3704
#4  0x00003fffaf7f3b60 in pmix_server_message_handler (pr=0x3fffa031e4f0, hdr=0x3fffa031ecb4, buf=0x3fffaebfe410, cbdata=0x0)
    at server/pmix_server.c:3750
#5  0x00003fffaf8c3a7c in pmix_ptl_base_process_msg (fd=-1, flags=4, cbdata=0x3fffa031ebe0) at base/ptl_base_sendrecv.c:784
#6  0x00003fffaf6ab918 in event_process_active_single_queue (base=0x10023d8d540, activeq=0x10023d415a0, max_to_process=2147483647, endtime=0x0)
    at event.c:1646
#7  0x00003fffaf6ac404 in event_process_active (base=0x10023d8d540) at event.c:1738
#8  event_base_loop (base=0x10023d8d540, flags=<optimized out>) at event.c:1961
#9  0x00003fffaf825d50 in progress_engine (obj=0x10023d8d4f0) at runtime/pmix_progress_threads.c:232
#10 0x00003fffaf488cd4 in start_thread () from /lib64/libpthread.so.0
#11 0x00003fffaf3b7e94 in clone () from /lib64/libc.so.6

I'm still digging.

jjhursey · 2020-01-29T23:00:19Z

bot:ibm:retest

jjhursey · 2020-01-29T23:07:36Z

bot:ibm:retest

rhc54 · 2020-01-29T23:13:16Z

How are you using IOF? For some reason, it appears you have asked PMIx to forward the stdin from some process - not clear what program you are running.

jjhursey · 2020-01-30T01:30:16Z

It's the cycle test, so it's just running prun -np 1 hostname over and over again. So the core is from prte forwarding stdin (which there shouldn't be any) from prun. Unless Jenkins is somehow injecting into the stream... It's strange.

I just confirmed that the --enable-debug build passes fine even under ci report here. But it seems to fail routinely in a non-debug build. However, outside of CI I've not been able to reproduce it manually. I'll keep trying to narrow it down.

bot:ibm:retest (back to a non-debg build)

jjhursey · 2020-01-31T22:05:25Z

bot:ibm:retest

jjhursey · 2020-02-04T19:41:15Z

bot:ibm:retest

* `hello_world` test which runs `hostname` and a Hello World PMIx client * `cycle` which tests running a bunch of jobs against the same PRRTE server set including a single node `hostname` and an init/finalize PMIx application across all slots. Signed-off-by: Joshua Hursey <jhursey@us.ibm.com>

jjhursey added the WIP Work In Progress (Do not merge) label Dec 20, 2019

jjhursey force-pushed the prrte-tests branch from b322347 to 74984dd Compare December 20, 2019 04:10

jjhursey commented Dec 20, 2019

View reviewed changes

prrte/hello_world/hello.c Show resolved Hide resolved

jjhursey commented Dec 20, 2019

View reviewed changes

prrte/cycle/run.sh Outdated Show resolved Hide resolved

jjhursey mentioned this pull request Dec 20, 2019

A CI Test vector openpmix/prrte#265

Closed

jjhursey mentioned this pull request Jan 14, 2020

OMPI integration issues to be resolved openpmix/prrte#298

Closed

8 tasks

jjhursey added the PRRTE_CI Trigger PRRTE CI on this PR label Jan 22, 2020

jjhursey force-pushed the prrte-tests branch from 4de6c4b to 649f7b4 Compare January 23, 2020 01:31

jjhursey force-pushed the prrte-tests branch from 4eb1c80 to 8dfed5d Compare January 28, 2020 21:57

jjhursey removed the WIP Work In Progress (Do not merge) label Jan 28, 2020

jjhursey force-pushed the prrte-tests branch from 2b0c873 to 48e9b09 Compare January 29, 2020 20:51

jjhursey marked this pull request as ready for review January 31, 2020 22:49

jjhursey force-pushed the prrte-tests branch from 48e9b09 to fdbc056 Compare January 31, 2020 22:50

jjhursey force-pushed the prrte-tests branch from fdbc056 to 2eddf99 Compare February 4, 2020 20:03

jjhursey force-pushed the prrte-tests branch from 2eddf99 to 7c8b450 Compare February 4, 2020 20:14

jjhursey merged commit 82d4f34 into openpmix:master Feb 4, 2020

jjhursey deleted the prrte-tests branch February 4, 2020 20:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Infrastructure for PRRTE CI Tests #49

Infrastructure for PRRTE CI Tests #49

jjhursey commented Dec 20, 2019

jjhursey commented Dec 20, 2019

jjhursey commented Jan 8, 2020

jjhursey commented Jan 9, 2020

jjhursey commented Jan 22, 2020

jjhursey commented Jan 22, 2020

jjhursey commented Jan 22, 2020

jjhursey commented Jan 23, 2020

jjhursey commented Jan 29, 2020

jjhursey commented Jan 29, 2020

jjhursey commented Jan 29, 2020

jjhursey commented Jan 29, 2020

jjhursey commented Jan 29, 2020

rhc54 commented Jan 29, 2020

jjhursey commented Jan 30, 2020

jjhursey commented Jan 31, 2020

jjhursey commented Feb 4, 2020

Infrastructure for PRRTE CI Tests #49

Infrastructure for PRRTE CI Tests #49

Conversation

jjhursey commented Dec 20, 2019

jjhursey commented Dec 20, 2019

jjhursey commented Jan 8, 2020

jjhursey commented Jan 9, 2020

jjhursey commented Jan 22, 2020

jjhursey commented Jan 22, 2020

jjhursey commented Jan 22, 2020

jjhursey commented Jan 23, 2020

jjhursey commented Jan 29, 2020

jjhursey commented Jan 29, 2020

jjhursey commented Jan 29, 2020

jjhursey commented Jan 29, 2020

jjhursey commented Jan 29, 2020

rhc54 commented Jan 29, 2020

jjhursey commented Jan 30, 2020

jjhursey commented Jan 31, 2020

jjhursey commented Feb 4, 2020