[ServiceBus] Improve auto lock renewer implementation

[yunhaoling]

addressing issue: #19362

currently sync auto lock renewer is implemented the way that each thread in the ThreadPoolExecutor handles lock renewal for one message/session at a time.

This could bring trouble when the amount of messages needed to be auto lock renewed is greater than the max workers of the executor -- E.g., executor has 4 workers, user register 10 messages, only the first 4 messages will be auto lock renewed.

This PR has proposed a new approach to support messages amount > max workers amount which is described as following:

The overall design is to decouple the renew process via a queue so that worker threads could keep popping and handling renew tasks from the queue.

The algorithm is:

• a: If users pass the max worker
  • a-1: if max worker is 1, then we have a single thread, which keeps pop renew task from -> check -> renew if needed -> push back renew tasks
  • a-2: if max worker is None or > 1, then the model is we have 1 dispatching thread, and (max_worker - 1) renew worker threads. The dispatching thread keeps popping task meta info from the queue and submit them to the executor. The worker thread do the renew work if needed and push the task meta info back into the queue if the object is still renewable.
• b: If users pass the executor, there's no public api in python to tell us the max worker value of an executor. we have to infer whether max_worker > 1 and the dispatching model could be applied. A inferring process will be started at the very beginning.
  • the inferring process would submit task within a submitted task, and then wait on the task for infer_timeout seconds. If max_worker > 1, then the sub-submitted task could finish,else waiting for the task would result in tineout error indicating that there's just 1 max_worker.
  • with the inferred max worker info, we could go to part a.

[yunhaoling]

/azp run python - servicebus - tests

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[yunhaoling]

/azp run python - servicebus - tests

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[yunhaoling]

/azp run python - servicebus - tests

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[yunhaoling]

/azp run python - servicebus - tests

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[yunhaoling]

/azp run python - servicebus - tests

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[yunhaoling]

/azp run python - servicebus - tests

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[yunhaoling]

/azp run python - servicebus - tests

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[yunhaoling]

/azp run python - servicebus - tests

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[annatisch]

LGTM :)

[yunhaoling]

/azp run python - servicebus - tests

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[yunhaoling]

/azp run python - servicebus - tests

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[swathipil]

lgtm!