New MPUBLISH command to publish multiple messages.

[roshkhatri]

From redis/redis#12213

This would enable publishing multiple messages on a single cluster bus message, avoiding the high cost incurred from the metadata overhead for a cluster bus message and increase the throughput.

We would also need a new type of message to indicate multiple messages (CLUSTERMSG_TYPE_MPUBLISH)

 MPUBLISH CHANNEL MESSAGE_1 MESSAGE_2 ...

It was previously given an OK by @madolson
Any thoughts?

[madolson]

@valkey-io/core-team I think this is straight forward enough to vote on. I believe the idea is to primarily reduce the overhead for clusterbus metadata. I suppose the alternative would be to implement the optimization that viktor mentioned in #307, where we try to avoid most of the overhead along the clusterbus and just send the publish metadata.

[zuiderkwast]

This is a new command AND a new cluster bus message.

I'd prefer #307 (comment) which is only a new cluster bus message.

How often does anyone publish multiple messages at the same time? Is it a real use case?

[madolson]

We had one person message it in Redis, you can see the attached link. I'm not aware of how popular it is.

[zuiderkwast]

OK, I've read the Redis issue now. If we add MPUBLISH, we're expected to add SMPUBLISH next?

Slight preference for multi-message PUBLISH over MPUBLISH. HMSET was deprecated in favor of multi-field HSET. Choosing MPUBLISH over multi-message PUBLISH seems inconsistent with that. The argument that we want to reserve the possibility to add more arguments doesn't seem to hold in this case. We can't add other future arguments to MPUBLISH with this syntax.

For the cluster bus, it seems logical to introduce just one new message that is lightweight and can carry multiple messages.

[madolson]

OK, I've read the Redis issue now. If we add MPUBLISH, we're expected to add SMPUBLISH next?

I think this was proposed because the overhead of using the PUBLISH command was too high in cluster mode. SMPUBLISH was thought to maybe get resolved by the other issue where we replicate SPUBLISH commands instead of pushing them over the clusterbus.

For the cluster bus, it seems logical to introduce just one new message that is lightweight and can carry multiple messages.

I think I'm also convinced this probably the right approach now.

[zuiderkwast]

Can we just add multi-PUBLISH channel message [ message ... ] now, together with #557, and get it over with? We haven't needed any flags or other args ti PUBLISH so far so I don't think it's necessary to reserve the possibility for that. We can just add a new command for that if/when that need comes.

[PingXie]

Can we just add multi-PUBLISH channel message [ message ... ] now, together with #557, and get it over with? We haven't needed any flags or other args ti PUBLISH so far so I don't think it's necessary to reserve the possibility for that. We can just add a new command for that if/when that need comes.

seems reasonable to me. It is a behavior change but not a breaking one IMO.

[hpatro]

#557 is close to the finish line, @zuiderkwast @PingXie I would prefer getting that merged and then have a separate PR to introduce the command related change. @roshkhatri should be able to help with it.

[PingXie]

No I am not advocating scope creep here. I was commenting on the issue itself without a timeline attached.

Speaking of #557, I actually would like us to think weigh the option of removing the complexity associated with the multi-message support.

[madolson]

Can we just add multi-PUBLISH channel message [ message ... ] now, together with #557, and get it over with? We haven't needed any flags or other args ti PUBLISH so far so I don't think it's necessary to reserve the possibility for that. We can just add a new command for that if/when that need comes.

seems reasonable to me. It is a behavior change but not a breaking one IMO.

I agree. My concern is we are adding features without good client support that nobody is likely to use. There are much more important features to work on that folks are asking for. The performance optimization is something that existing users benefit from.

I'm concerned generally that we are accepting too many command modifications without really thinking through if they will be useful long term.