[core][distributed] add message queue for cross-node broadcast

[youkaichao]

after we move all the input to CPU, this will help move the driver process outside of the tp 0 worker.

TODO: add multi-node test.

[cadedaniel]

I'm supportive of this, although I think we should ensure cross-node operations API are compatible with Ray. otherwise we will have to implement a long tail of features to get fault tolerance / handle esoteric TCP errors. 🤮

[youkaichao]

cross-node operations API are compatible with Ray

what does this mean?

[cadedaniel]

if users want fault tolerance, it should be easy to use Ray for that. we could implement it ourselves if we want so people don't have to use Ray, but it's a lot of work

[youkaichao]

turns out shm broadcast is more efficient in single node:

shm broadcast

Throughput: 39.08 requests/s, 20006.60 tokens/s

pyzmq broadcast:

Throughput: 37.61 requests/s, 19256.93 tokens/s

so we should use shm broadcast inside single node, and use pyzmq for cross-node oly.