stream-buffering is an OMPI option, so make it an MCA

[jjhursey]

• It looks like --stream-buffering CLI was copied from the OMPI schizo
  to others, but is not processed by others.
• In OMPI make this an MCA parameter, and translate the --stream-buffering
  to that MCA parameter.

[rhc54]

I'm not sure this is going to work due to the DVM. This was my question - does stream buffering have to be set at time of DVM start? Or is it something that can be set during operation?

Of course, I see no way to do it on a per-job basis - just trying to figure out how to present this to the user.

[jjhursey]

It has to be set inside the application. So it makes sense that it is being set inside the OMPI library.
So whether starting with prun or prterun as long as the MCA parameter is being passed then we are fine (similar to the other MCA options in the schizo/ompi).

Pairs with: open-mpi/ompi#10746

[jjhursey]

Note that I removed the CLI option from the other PRRTE tools because they didn't have the backing functionality to do anything with it. So it was a dead option. The OMPI schizo was the only one that checked for it and did something.

[rhc54]

I'm trying to wrap my head around this. If you set it in the app, then the buffering to the prted pipes is changed, but nothing else. In particular, the buffering at the output end of mpirun is unaffected. Thus, it isn't clear to me that the user is going to see what they expected to see.

I checked and it looks to me like this does recover the old OMPI behavior, and I agree that OMPI is the only one who will do anything with it. MCA param vs cmd line option is up to you guys. I'd just like to understand what the user thinks is supposed to happen when they set this so we know that we are really meeting expectations.

[jjhursey]

The motivation for the change was for an environment where the default was a fully buffered stdout/stderr. So an application could run to completion before any output was flushed from the application process to whatever was collecting it (or see it it come in big chunks over time). This was probably done to reduce the IO burden over the life of a batch job where no one is watching it live - I'm not sure it was an old Cray system. The motivation of the change was to add an option to libmpi in the application to change the default behavior (vs updating the application to make this call).

That is not the default behavior on systems according to man setvbuf

If a stream refers to a terminal (as stdout normally does) it is line buffered. The standard error stream stderr is always unbuffered by default.

[rhc54]

Understood - but do we not then need to apply that setting (or something like it) to mpirun as well? The IO will simply collect there and be emitted in chunks, I think - I don't recall if the PMIx IOF code gets around it, but perhaps you've looked at it and concluded it is okay?

I remember seeing this problem when testing stdin forwarding. I can't recall if we hit it on the input end, or if the output (where we simply parroted what came into the test process) was getting chunked.

[rhc54]

Just to be clear: I have no issue with committing this change - I'm only asking these questions because I want to ensure we wind up with the desired behavior.

[jjhursey]

This sounds like a feature request - expanding the steam buffering controls into PRRTE/PMIx. Sounds like an interesting thing to explore.

Unfortunately, I don't have the cycles to commit to that exploration in the scope of this PR and the Open MPI v5 release timeline. I was asked by the OMPI RMs to fix the stream buffering as is in OMPI, which this PR and the corresponding OMPI PR do.

The MCA option was meant as a developer diagnostic tool that I think few users ever use (I do use it occasionally, and have confirmed that it works as originally intended). I'd actually like to deprecate the CLI option. I've added a commit that adds the deprecation warning for this option.

[rhc54]

okay 🤷‍♂️ I agree this is probably rarely used, so no point wasting time on it.

[rhc54]

bot:ibm:retest