queue: support priority queue #72

glorv · 2022-12-02T02:13:25Z

this PR introduces a new type of queue: priority queue. The priority queue sorts task by their priority value that can allow inserting tasks in the front or middle, so we can support a more flexible schedule policy.

Why Priority

The default multi-level queue is suitable for most FIFO scenarios. But in cases that need more fine-grained control of the task schedule order, FIFO is not good enough. For example, in a shared environment for multiple users, the scheduler should schedule tasks from different users according to their share. In this case, a priority-based scheduler based on the total execution time by the user may be a workable solution.

Drawbacks

The priority queue is not perfect for all cases. The scheduling overhead is a little bigger than the multi-level queue. And there is no default implementation that is suitable for common cases. The schedule also doesn't implement the multi-level strategy that automatically degrades slow tasks.

Implement detail

The priority queue is backed by crossbeam-skiplist, the key is the priority value and the value is the task. The implementation attaches the key with an extra auto-incremented sequence number to ensure the key never duplicate with each other.

The user should provide an implementation of the trait TaskPriorityPriovider to generate the priority value of each task.

Benchmark Result

I test the priority queue performance by comparing the overall performance of tidb cluster with sysbench oltp_read_only and select_random_ranges workload.
In this brief benchmark, the priority scheduler implements the mclock algorithm to support fair scheduling among multi resource groups. Becuase by default there is only one resource group, so it can be treat as a FIFO queue. The priority test code is: https://github.com/glorv/tikv/tree/qos-http and the nightly code is commit 2704588c6aaa1a269bb91499229e358d23cc636b.

The benchmark is running on 3 * 8core tikv and 6 * 8 core tidb so tikv should be the performance bottleneck.
Result:

(the first is priority queue and second is yatp multi-level queue)

The benchmark shows that the overhead of priority-queue is bigger than multi-level queue. The performance drops about 5% in oltp_read_only and 1% in select_random_ranges.

Signed-off-by: glorv <glorvs@163.com>

glorv · 2022-12-02T02:13:55Z

@Connor1996 @sticnarf @BusyJay PTAL

Signed-off-by: glorv <glorvs@163.com>

BusyJay · 2022-12-06T18:26:06Z

Can you highlight the cases that priority queue solves better than multi level queue? And what's the drawbacks of priority queue?

glorv · 2022-12-07T03:48:58Z

Can you highlight the cases that priority queue solves better than multi level queue? And what's the drawbacks of priority queue?

@BusyJay I have changed the description, PTAL again, thanks

Signed-off-by: glorv <glorvs@163.com>

glorv · 2022-12-13T02:10:16Z

@Connor1996 @sticnarf PTAL, thanks

sticnarf · 2022-12-13T10:00:19Z

Cargo.toml

@@ -10,6 +10,7 @@ repository = "https://github.com/tikv/yatp/"

 [dependencies]
 crossbeam-deque = "0.8"
+crossbeam-skiplist = { git = "https://github.com/crossbeam-rs/crossbeam" }


crossbeam-skiplist 0.1 recently releases. We can use the crates.io version now.

sticnarf · 2022-12-13T10:15:38Z

src/queue/priority.rs

+    }
+}
+
+/// A holder to store task. The slot can be concurrently visit by multiple thread in the skip-list.


I only see push and pop access the skiplist and they don't seem to visit a slot concurrently. Where does concurrent visit happen?

Yes, in our case it's safe to take the value. But the pop() api only provide readonly access to the key and value. So here we wrap it in a atomic ptr to workaround this restriction. Do you have any idea to avoid this?

Ah, I see. The skiplist cannot guarantee that the entry is not borrowed by others when popped, so we only have read only access to the value.

Instead of an AtomicPtr wrapping a Box, do you think a crossbeam AtomicCell is simpler and cheaper? There will be no extra allocation. The future TaskCell is just a pointer so it should also be lock free.

Seems https://github.com/crossbeam-rs/crossbeam/blob/0de660ec88481ff7787374efee0c2e21d4d403b2/crossbeam-utils/src/atomic/atomic_cell.rs#L1020 AtomicCell is also not very cheap because it uses a global lock for large value (> 8 bytes).

Ok. I found we can simply replace the AtomicPtr with RefCell<Option<T>>. Since only the thread pops the entry can visit the value, it's safe to impl Sync for the Slot type. @sticnarf PTAL again.

The future taskcell is only 8 bytes so atomic cell is cheap. For other types, yes, a Mutex is better than the implementation of AtomicCell.

While I don't think RefCell + unsafe impl Sync is a good combination.

If you care about safety, Mutex<Option<>> is your choice. And it's fast in practice when there is not any race (only need a very few atomic operations).
If you are confident enough, just use UnsafeCell. RefCell does not provide you with extra safety when you mark it Sync because its borrow checks will become unreliable when multiple threads access it concurrently.

Done. Change to using AtomicCell. Though UnsafeCell is also an available choice, since this isn't a performance bottleneck, using AtomicCell can avoid unsafe code.

sticnarf · 2022-12-13T10:16:11Z

src/queue/priority.rs

+///
+#[derive(Clone)]
+struct PriorityTaskManager {
+    level_manager: Arc<TaskLevelManager>,


With a priority queue, why is there still a level manager?

While using the priority queue, we still need to auto demote slow tasks. So the priority manager can use the level information to set different priority value. Or we have to implement similar mechanism to do so.

Signed-off-by: glorv <glorvs@163.com>

sticnarf

LGTM.
@BusyJay Do you want to take another look?

Connor1996 · 2022-12-20T19:14:22Z

src/pool/builder.rs

@@ -290,6 +290,17 @@ impl Builder {
        self.build_with_queue_and_runner(QueueType::Multilevel(queue_builder), runner_builder)
    }

+    ///


Connor1996 · 2022-12-20T19:15:16Z

src/queue/extras.rs

@@ -28,6 +28,8 @@ pub struct Extras {
    pub(crate) fixed_level: Option<u8>,
    /// Number of execute times
    pub(crate) exec_times: u32,
+    /// The task group id. Used in priority queue.
+    pub(crate) group_id: u64,


why not group name?

Done. But I think it a bit weird here because the group_id/name is only fits in some specified cases(e.g. in tikv), but may not fit for other cases. I'm not sure if it is better to be defined as extra_data: Vec<u8>, so the user can put arbitary data in this field as they need.

Signed-off-by: glorv <glorvs@163.com>

nolouch · 2022-12-21T03:32:41Z

src/queue/priority.rs

+    }
+
+    #[inline]
+    fn gen_key(&self, weight: u64) -> MapKey {


may s/weight/prioriy/ more unify.

nolouch · 2022-12-21T03:41:29Z

src/queue/priority.rs

+}
+
+impl Config {
+    /// Sets the name of the multilevel task queue. Metrics of multilevel


Suggested change

/// Sets the name of the multilevel task queue. Metrics of multilevel

/// Sets the name of the priority task queue.

nolouch · 2022-12-21T03:43:28Z

src/queue/priority.rs

+}
+
+impl Builder {
+    /// Creates a multilevel task queue builder with specified config and [`TaskPriorityProvider`].


Suggested change

/// Creates a multilevel task queue builder with specified config and [`TaskPriorityProvider`].

/// Creates a prioriy task queue builder with specified config and [`TaskPriorityProvider`].

nolouch · 2022-12-21T03:55:49Z

src/queue/priority.rs

+/// The configurations of priority task queues.
+pub struct Config {
+    name: Option<String>,
+    level_time_threshold: [Duration; LEVEL_NUM - 1],


can't configure this field?

It's the same as multi-level, not configurable currently

BusyJay · 2022-12-21T03:57:52Z

src/queue.rs

@@ -18,7 +19,7 @@ pub use self::extras::Extras;
 use std::time::Instant;

 /// A cell containing a task and needed extra information.
-pub trait TaskCell {
+pub trait TaskCell: 'static {


The skiplist's value must be 'static. I think it's fair that a task spawned in the thread pool should live long enough.

I don't see why skiplist's value should be 'static. Technically, as long as the value outlive skiplist, it should be OK. If that's an API shortcoming, then the constraint should be put into priority queue instead of here.

It looks like an API shortcoming. I tried making the constraint priority-queue-only, but because we use enum to dispatch instead of trait, the constraint propagates up.

BusyJay · 2022-12-21T03:58:38Z

src/queue/extras.rs

@@ -28,6 +28,8 @@ pub struct Extras {
    pub(crate) fixed_level: Option<u8>,
    /// Number of execute times
    pub(crate) exec_times: u32,
+    /// The task group name. Used in priority queue.
+    pub(crate) group_name: String,


I don't see it's set or get in any place.

In the multi-tenant implementation, we need to attach the task with the resource group name. I change this field to metadata: Vec<u8> to allow priority queue user to store arbitrary data here.

Connor1996 · 2022-12-21T04:38:48Z

src/pool/builder.rs

+    ///
+    pub fn build_priority_future_pool(
+        &self,
+        priority_priovider: Arc<dyn priority::TaskPriorityProvider>,


priovider -> provider

Connor1996 · 2022-12-21T04:43:19Z

src/queue/priority.rs

+        if let Some(ref running_time) = total_running_time {
+            running_time.inc_by(elapsed);
+        }
+        if level == 0 {


shouldn't always be 0?

No. The priority runner also implements the level degradation logic like the multilevel queue, so the user may assign a different priority for higher level tasks.

Connor1996 · 2022-12-21T04:46:41Z

src/queue/priority.rs

+    fn get_priority(&self, extras: &Extras) -> u64;
+}
+
+///


Signed-off-by: glorv <glorvs@163.com>

glorv · 2022-12-21T06:50:02Z

@BusyJay @Connor1996 @nolouch PTAL again, thank you very much.

Connor1996 · 2022-12-21T18:52:30Z

src/queue/extras.rs

+        &mut self.metadata
+    }
+
+    /// Set the group name of this task.


outdated comment

Signed-off-by: glorv <glorvs@163.com>

BusyJay · 2022-12-23T06:18:29Z

src/queue/priority.rs

+    }
+}
+
+/// The local queue is just a proxy of the global queue.


Then why not just use the same struct? You can define different alias.

BusyJay

How about updating benches to include microbench of the new queue?

Rest LGTM

BusyJay · 2022-12-23T06:42:55Z

src/queue/priority.rs

+pub trait TaskPriorityProvider: Send + Sync + 'static {
+    /// Return a priority value of this task, all tasks in the priority
+    /// queue is ordered by this value.
+    fn get_priority(&self, extras: &Extras) -> u64;


Suggested change

fn get_priority(&self, extras: &Extras) -> u64;

fn priority_of(&self, extras: &Extras) -> u64;

nolouch

lgtm

Connor1996

LGTM

Signed-off-by: glorv <glorvs@163.com>

BornChanger · 2022-12-26T13:59:01Z

/test-address-sanitizer-ubuntu-latest-nightly

glorv · 2022-12-28T05:56:03Z

@BusyJay PTAL again

BornChanger

LGTM

glorv · 2022-12-29T06:52:08Z

The failed case is caused by the extra check by Miri, and is fixed by this PR crossbeam-rs/crossbeam#940. Since it doesn't affect correctness, we can fix in the next release version of crossbeam-skiplist.

glorv · 2022-12-29T07:15:43Z

The failed case is caused by the extra check by Miri, and is fixed by this PR crossbeam-rs/crossbeam#940. Since it doesn't affect correctness, we can fix in the next release version of crossbeam-skiplist.

crossbeam-skip already released v0.1.1 which included the fixed PR, so we can just rerun the failed case.

support priority queue

1b9d2d4

Signed-off-by: glorv <glorvs@163.com>

glorv requested review from sticnarf, BusyJay and Connor1996 December 2, 2022 02:14

fix test

d7dc51b

Signed-off-by: glorv <glorvs@163.com>

optimize the implement of task slot

5e35595

Signed-off-by: glorv <glorvs@163.com>

sticnarf reviewed Dec 13, 2022

View reviewed changes

glorv added 2 commits December 14, 2022 10:40

use crates.io

4bde2f5

Signed-off-by: glorv <glorvs@163.com>

replace AtomicPtr with RefCell

c652691

Signed-off-by: glorv <glorvs@163.com>

glorv force-pushed the priority branch from cea45e3 to c652691 Compare December 15, 2022 16:31

change RefCell with AtomicCell

dd767d2

Signed-off-by: glorv <glorvs@163.com>

sticnarf previously approved these changes Dec 20, 2022

View reviewed changes

Connor1996 reviewed Dec 20, 2022

View reviewed changes

use group name in extras

4a056c4

Signed-off-by: glorv <glorvs@163.com>

glorv dismissed sticnarf’s stale review via 4a056c4 December 21, 2022 02:05

nolouch reviewed Dec 21, 2022

View reviewed changes

BusyJay reviewed Dec 21, 2022

View reviewed changes

Connor1996 reviewed Dec 21, 2022

View reviewed changes

src/queue/priority.rs Outdated

fn get_priority(&self, extras: &Extras) -> u64;

}

///

Copy link

Member

Connor1996 Dec 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

comment

resolve comments

298e50e

Signed-off-by: glorv <glorvs@163.com>

glorv force-pushed the priority branch from 055f707 to 298e50e Compare December 21, 2022 06:48

Connor1996 reviewed Dec 21, 2022

View reviewed changes

src/queue/extras.rs Outdated

&mut self.metadata

}

/// Set the group name of this task.

Copy link

Member

Connor1996 Dec 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

outdated comment

fix a comment

c9c1050

Signed-off-by: glorv <glorvs@163.com>

nolouch mentioned this pull request Dec 22, 2022

Global Resource Quota Control pingcap/tidb#38825

Closed

41 tasks

BusyJay reviewed Dec 23, 2022

View reviewed changes

nolouch reviewed Dec 23, 2022

View reviewed changes

glorv force-pushed the priority branch from 04284b6 to c629e34 Compare December 25, 2022 15:08

BornChanger previously approved these changes Dec 26, 2022

View reviewed changes

Connor1996 previously approved these changes Dec 26, 2022

View reviewed changes

fix comments and add benchmark

9dcf235

Signed-off-by: glorv <glorvs@163.com>

glorv dismissed stale reviews from Connor1996 and BornChanger via 9dcf235 December 26, 2022 08:50

glorv force-pushed the priority branch from c629e34 to 9dcf235 Compare December 26, 2022 08:50

glorv mentioned this pull request Dec 28, 2022

*: add resource group for the read path tikv/tikv#14001

Merged

BornChanger approved these changes Dec 29, 2022

View reviewed changes

Connor1996 approved these changes Dec 29, 2022

View reviewed changes

BusyJay merged commit bcf431a into tikv:master Dec 29, 2022

glorv deleted the priority branch December 29, 2022 07:23

Connor1996 mentioned this pull request Jan 29, 2023

batch-system: add priority scheduling for batch system tikv/tikv#14065

Merged

	/// Sets the name of the multilevel task queue. Metrics of multilevel
	/// Sets the name of the priority task queue.

	/// Creates a multilevel task queue builder with specified config and [`TaskPriorityProvider`].
	/// Creates a prioriy task queue builder with specified config and [`TaskPriorityProvider`].

	fn get_priority(&self, extras: &Extras) -> u64;
	fn priority_of(&self, extras: &Extras) -> u64;

queue: support priority queue #72

queue: support priority queue #72

Conversation

glorv commented Dec 2, 2022 • edited Loading

Why Priority

Drawbacks

Implement detail

Benchmark Result

glorv commented Dec 2, 2022

BusyJay commented Dec 6, 2022

glorv commented Dec 7, 2022

glorv commented Dec 13, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

glorv Dec 15, 2022 • edited Loading

Choose a reason for hiding this comment

glorv Dec 15, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

glorv Dec 16, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sticnarf left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nolouch Dec 21, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

glorv commented Dec 21, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BusyJay left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nolouch left a comment

Choose a reason for hiding this comment

Connor1996 left a comment

Choose a reason for hiding this comment

BornChanger commented Dec 26, 2022

glorv commented Dec 28, 2022

BornChanger left a comment

Choose a reason for hiding this comment

glorv commented Dec 29, 2022

glorv commented Dec 29, 2022

glorv commented Dec 2, 2022 •

edited

Loading

glorv Dec 15, 2022 •

edited

Loading

glorv Dec 15, 2022 •

edited

Loading

glorv Dec 16, 2022 •

edited

Loading

nolouch Dec 21, 2022 •

edited

Loading