rdma: do local RDMA read on all NIC rails for flush()

[taeilum00]

This changes flush() implementation in such a way that local RDMA read is requested to all NIC rails for multi-rail cases. This is to make sure all data traffic paths are safely flushed from all NICs to GPU.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[bwbarrett]

The last sentence in your patch is misleading. The flush function is never disabled. It's just not called in some situations.

[ghost]

Please drop the merge commit and squash your fixups.

A rebased branch is here: https://github.com/aws-nslick/nccl-net-ofi/tree/taeil-flush-rebased

You can do this locally by just calling:

git checkout flush
git remote add upstream git@github.com:aws/aws-ofi-nccl.git
git remote update --prune
git branch --set-upstream-to=upstream/master
git rebase --signoff -i upstream/master
git push --force-with-lease origin flush

[bwbarrett]

Other than the two small nits, looks good to me.