Question: Formula in CorkDispatcher

[zhant09]

We are working on an issue similar to micro-batching task in BentoML. I know the CorkDispatcher class implements the cork algorithm, but I don't get the meaning of below code. What does the calculation mean? Any help would be appreciated.

https://github.com/bentoml/BentoML/blob/master/bentoml/marshal/dispatcher.py#L185

if n * (wn + dt + (a or 0)) <= self.optimizer.wait * decay:
    await asyncio.sleep(self.tick_interval)
    continue

[bojiang]

In short, it compares the mathematical expectations of the overall waiting time of each requests under the two decisions cork and release now.

[zhant09]

Thank you for the reply, I can feel that overall meaning now. But I still have some question about the calculation detail.

For example, like in current code settings, a = 2, the condition would be met only when n is 0. Why is that?

Some related question...
What is the measuring unit of a, is it millisecond?
Why there is divide 1000 calculation for self.max_latency_in_ms? Does it converted to time in second?

self.max_latency_in_ms = max_latency_in_ms / 1000.0

https://github.com/bentoml/BentoML/blob/master/bentoml/marshal/dispatcher.py#L112

[zhant09]

I have read more detail of the code and discussed with colleague, and I could answer part of my own question now.

1. The a or self.optimizer.o_a is an param that would be optimized during running process using least squares method, which explained Adaptive micro-batching.
2. The measuring unit of a and self.max_latency_in_ms is second.

But I still have question about this formula, why there is an n*wn?

n * (wn + dt + (a or 0)) <= self.optimizer.wait * decay

[bojiang]

wn represents the time that the last request in queue has waited for. The 'cork' decision drags down all the requests in queue, so the loss should be n * wn.

[zhant09]

Thank you for the reply.

Then from my understanding, n * a means the overall process time (consumer running duration), n * dt is the overall tick interval if sleep happens, wn is the latest request's waiting time, we regard n * wn as the overall queue waiting time.

So for left side, it's overall time consuming for this batch. But why does this only compare to self.optimizer.wait? This seems to be only average queue waiting time?

[zhant09]

Same question here:
#927 (comment)