Fix[MQB]: use CSL to update state on QueueAssignmentAdvisory

[emelialei88]

Similar to PR #581, this PR invokes CSL commit callback for QueueAssignmentAdvisories and the queue assignment part of LeaderAdvisory, effectively making CSL the source of truth.

[kaikulimu]

Some comments

[kaikulimu]

Looks correct for two reasons:

1. Should be QueueAssignmentAdvisory instead of QueueUnAssignmentAdvisory
2. This log line seems to be during

   blazingmq/src/groups/mqb/mqbc/mqbc_clusterutil.cpp

   Lines 1000 to 1003 in 7b30704

   	// Apply 'queueAssignmentAdvisory' to CSL
   	BALL_LOG_INFO << clusterData->identity().description()
   	<< ": 'QueueAssignmentAdvisory' will be applied to "
   	<< " cluster state ledger: " << queueAdvisory;

   , but should appear when we go through CSL path.

[kaikulimu]

Now, QueueAssignmentAdvisory should never be sent out. We should remove

blazingmq/src/groups/mqb/mqbblp/mqbblp_cluster.cpp

Lines 3079 to 3087 in 7b30704

	case MsgChoice::SELECTION_ID_QUEUE_ASSIGNMENT_ADVISORY: {
	dispatcher()->execute(
	bdlf::BindUtil::bind(
	&ClusterOrchestrator::processQueueAssignmentAdvisory,
	&d_clusterOrchestrator,
	message,
	source),
	this);
	} break; // BREAK

and

blazingmq/src/groups/mqb/mqbblp/mqbblp_clusterorchestrator.cpp

Line 1341 in 7b30704

void ClusterOrchestrator::processQueueAssignmentAdvisory(

and

blazingmq/src/groups/mqb/mqbi/mqbi_clusterstatemanager.h

Lines 336 to 339 in 7b30704

	virtual void
	processQueueAssignmentAdvisory(const bmqp_ctrlmsg::ControlMessage& message,
	mqbnet::ClusterNode* source,
	bool delayed = false) = 0;

.

[kaikulimu]

Still, preserve enough code such that mqbblp::ClusterStateManager::processBufferedQueueAdvisories() and mqbblp::ClusterStateManager::processLeaderAdvisory() do not break.

[kaikulimu]

Thinking about compatibility between broker versions, we should still broadcast this advisory for now, such as old broker can still receive and process.

[emelialei88]

When the new leader broadcast through applyRecordInternal, if the followers are old version, it seems that they can still receive this e_CLUSTER_TYPE event and process correctly. Could you given an example of where there could be a mismatch?

[kaikulimu]

What I meant is: Old version followers return on commit callback of e_CLUSTER_TYPE events, so they rely on having this additionally broadcasted 'queueAssignmentAdvisory' to process correctly. Thus, new version leader should keep broadcasting this additional time for now,

[kaikulimu]

Thinking about compatibility between different broker versions, we still keep this case as an no-op, in case an old broker sends us this:

case MsgChoice::SELECTION_ID_QUEUE_ASSIGNMENT_ADVISORY: {
    // NO-OP
} break; // BREAK

[emelialei88]

Yes, and upon some thinkings, maybe we should keep this branch and not remove anything related to processQueueAssignmentAdvisory for now. Consider this: when the old leader calls assignQueue, it will broadcast SELECTION_ID_QUEUE_ASSIGNMENT_ADVISORY. If we delete this case for new followers, they won't do anything upon receiving the advisory which is incorrect.

[kaikulimu]

Old leader will also write QAA to CSL; old followers will return immediately, but new followers should process QAA as part of CSL commit callback. It should be safe to remove this branch.