Task channel aware rescheduling

[yycptt]

What changed?

• Make task rescheduler aware of task channel assignment and stop rescheduling if task channel is full.
• Move task priority assigning into task executable and do it before adding to rescheduler since doing it upon rescheduling may change the task channel assignment.
• Code refactoring
  • Remove error from the return value of history task Scheduler interface
  • Combine the impl of fifo scheduler and namespace priority scheduler
  • some misc. renaming

Why?

• Existing impl of task rescheduler always pop all tasks that are due for rescheduling since all tasks are in one priority queue, and even if one task can't be submitted, we don't know about the next one. This will consume tons of cpu resource if most tasks in the rescheduler are from one namespace and the task channel for that namespace is always full.

How did you test it?

• Tested on test cluster. Reduced cpu usage of rescheduler from ~80% to almost 0.

Potential risks

• The number of priority queue may potential be large. But as long as namespace are not active on all shard at the same time (very unlikely) it should be ok.

Is hotfix candidate?

• No.

[yiminc]

maybe worth to consider object pooling if profiling shows frequent create/destroy pq become a bottleneck.

[yycptt]

yeah make sense.

I am also thinking that maybe this check can be moved to the beginning of the loop, so we don't immediately clean up the pq when it's empty but wait for the next round to see if it's still empty. If there's no next round, storing some empty pq seems ok to me?

[yycptt]

I updated the logic to clean up empty PQ every 3 minutes to avoid keep creating & deleting pq, especially for scheduled queue.

I am going to land this PR. Let me know if you have any concern re. ^ change. If so, I am happy to change it in a later pr. Thanks.