Fix OSC RDMA to work in cases where one BTL cannot reach all communicator group members

[jsquyres]

Cisco MTT shows that MPI_WIN_CREATE is failing since the OSC pt2pt component was removed (i.e., MPI_WIN_CREATE fails because there is no OSC component available). Here's an example of a simple test failing because MPI_WIN_CREATE failed.

After discussion on the 16 June 2020 weekly Open MPI webex, @hjelmn said that he could add a few functions to make OSC RDMA work in all cases (not just in cases where a single BTL can reach all members in the communicator group).

Marking this as a blocker for v5.0 because it's preventing MPI_WIN_CREATE from working in at least some environments, and because -- per the discussion between @hjelmn and @bwbarrett -- the fix to OSC RDMA doesn't seem too difficult.

[hjelmn]

I have the implementation mostly complete. It will require reworking how BTL receive completions are handled but that is a good thing. Been looking at reworking those for years. I have a branch with the changes but I can not test usnic or portals4. I verified vader, tcp, and self. Will validate uGNI and UCX today.

[jsquyres]

Send me the branch and what tests to run, and I'll check usnic.

[hjelmn]

https://github.com/hjelmn/ompi/tree/btl_base_atomics_are_awesome

This only has the changes to the BTL receive logic. Please run a standard set of two-sided tests. IMB would work. Feel free to push back and changes btl/usnic needs.

The old receive callback got btl, tag, descriptor, cbdata. The new callback gets the btl and a new receive descriptor which contains the tag, endpoint, received message, and cbdata.

[hjelmn]

Really curious if there are any performance regressions. There shouldn't be but that is going to be where any pushback will be.

[hjelmn]

@jsquyres Ping. Please take a look at the branch mentioned above. I need to know if this change works for you as I can not verify the usnic BTL changes. I want to get this change approved and in this week so I can get the btl base atomics support finished.

[jsquyres]

@hjelmn Sorry for the delay. I ran the https://github.com/hjelmn/ompi/tree/btl_base_atomics_are_awesome branch last night and things generally look good. I'm getting some random failures, but I'm apparently getting some random failures with master, too. Specifically: with the btl_base_atomics_are_awesome branch, I'm seeing most (all?) of the one-sided tests that were failing are now passing.

We'll have to address the other random failures in another issue.

[jsquyres]

This is still causing terabytes of corefiles with Cisco's MTT that I regularly have to manually clear out to avoid filling my filesystem.

I'm going to have to disable Cisco MTT's one-sided testing until this is fixed.

[jsquyres]

I don't know if this is related to #7892 or not, but I figured I'd put in the cross-reference.