Add ConcurrentExecution step

[gtopper]

ML-6015

Based on the preexisting internal _ConcurrentJobExecution class. With support for three concurrency mechanisms: asyncio, threading, and multiprocessing.

[alxtkr77]

Aside from the issue with serializable context, everything else in the pull request looks good. Do we want to address this issue within the current pull request? It seems that the 'pass_context' flag may not be the best solution for the problem.

[gtopper]

Do we want to address this issue within the current pull request?

To make mlrun context serializable isn't something we can do in storey anyway.

It seems that the 'pass_context' flag may not be the best solution for the problem

Why is it the flag's fault? :) Do you have some change in mind?

[assaf758]

Do we want to address this issue within the current pull request?

To make mlrun context serializable isn't something we can do in storey anyway.

It seems that the 'pass_context' flag may not be the best solution for the problem

Why is it the flag's fault? :) Do you have some change in mind?

Can you check if indeed https://github.com/uqfoundation/dill can serialize lambdas?

[gtopper]

@assaf758, it can, but it's not obvious how to use that to solve the problem. I'll try to find a way.

[gtopper]

It turns out that mlrun GraphContext contains the entire graph, which even dill fails to serialize. Ironically, it's there for ParallelRun.