[FEA] Need a way to block on segment initialization

[dagardner-nv]

Is your feature request related to a problem? Please describe.
In Morpheus we need a way to block until the segment initializer has been called.
nv-morpheus/Morpheus#188

Currently Morpheus is building the pipeline with:

self._srf_pipeline.make_segment("main", inner_build)

self._srf_executor.register_pipeline(self._srf_pipeline)

self._is_built = True

The problem is that inner_build is called asynchronously and we need a way to wait until after inner_build is called.

Describe the solution you'd like
Probably some sort of blocking call that we can make like:
wait_for_segment_init

Additional context
In Morpheus we don't have the full detail of the messages passed between nodes until the pipeline is fully built, which doesn't happen until the segment initialize is called.

[ryanolson]

It is not safe to assume that every segment defined in a pipeline will be instantiated.

If the primary motivation here is to generate a viz file, we should be able to do it from the pipeline definition and proto serialization; however, in the case of python, we not be able to generate full type information.

[dagardner-nv]

@ryanolson doesn't calling Executor.start() require that all the segment initializers are called?

[ryanolson]

For the standalone / single node executor, that is indeed what happens.

But it is not generally true. If you had a two segment, two machine pipeline with segment A on machine A and segment B on machine B, the local executor would only ever construct/initialize A or B.

[dagardner-nv]

From Morpheus' perspective then, we would always know which specific segment we are interested in. So in the above example:

self._srf_pipeline.make_segment("main", inner_build)
self._srf_executor.register_pipeline(self._srf_pipeline)

self._srf_pipeline.wait_for_segment_init("main")
self._is_built = True

[mdemoret-nv]

I agree with @ryanolson. You never know if the initializer is going to be called or not. The only thing you can do is await on start().

I think we should close this issue as wontfix. The better solution is to handle this in morpheus with nv-morpheus/Morpheus#84