Support for Decouping of Compute and Storage of TiFlash

[guo-shaoge]

Feature Request

Describe your feature request related problem

Describe the feature you'd like

The feature requests are from the Decoupling of Compute and Storage for TiFlash.

• Add new label for TiFlash ReadNode: Now the label of TiFlash is <engine, tiflash>, we need to to add a new lable <engine, mpp> for tiflash nodes which only used for MPP computation(aka ReadNode). And the region data will be stored in WriteNode.
• Add node cache information(like region cache), then TiDB use node information when dispatch MPPTask.

Describe alternatives you've considered

Teachability, Documentation, Adoption, Migration Strategy

[JmPotato]

IMO, we can use Placement Rules to keep data from being relocated to a ReadNode. For example, we can have a rule like this:

{
  "group_id": "tiflash",
  "id": "learner-replica-table-test",
  "start_key": "7480000000000000ff2d5f720000000000fa",
  "end_key": "7480000000000000ff2e00000000000000f8",
  "role": "learner",
  "count": 2,
  "label_constraints": [
    {"key": "engine", "op": "in", "values": ["tiflash"]}
  ],
  "location_labels": ["host"]
}

Due to the <engine, mpp> label, there is no region that will be moved to the ReadNode.

As for the region cache dispatch, client-go supports using the store label to distinguish different request targets already. You can just re-use these parts of code.

https://github.com/tikv/client-go/blob/e31ba21ae0b24ba0863445f9eefba4a8cabedf89/internal/locate/region_cache.go#L522

https://github.com/pingcap/tidb/blob/36c348899fc9d82db8760ad1ddf33401fb7ecf14/distsql/request_builder.go#L56

[guo-shaoge]

IMO, we can use Placement Rules to keep data from being relocated to a ReadNode. For example, we can have a rule like this:

{
  "group_id": "tiflash",
  "id": "learner-replica-table-test",
  "start_key": "7480000000000000ff2d5f720000000000fa",
  "end_key": "7480000000000000ff2e00000000000000f8",
  "role": "learner",
  "count": 2,
  "label_constraints": [
    {"key": "engine", "op": "in", "values": ["tiflash"]}
  ],
  "location_labels": ["host"]
}

Due to the <engine, mpp> label, there is no region that will be moved to the ReadNode.

As for the region cache dispatch, client-go supports using the store label to distinguish different request targets already. You can just re-use these parts of code.

https://github.com/tikv/client-go/blob/e31ba21ae0b24ba0863445f9eefba4a8cabedf89/internal/locate/region_cache.go#L522

https://github.com/pingcap/tidb/blob/36c348899fc9d82db8760ad1ddf33401fb7ecf14/distsql/request_builder.go#L56

Thanks! Really helpful.

[guo-shaoge]

Set label for tiflash: pingcap/tiflash#4779
Find tiflash stores with mpp label: tikv/client-go#481

[JaySon-Huang]

IMO, we can use Placement Rules to keep data from being relocated to a ReadNode. For example, we can have a rule like this:

{
  "group_id": "tiflash",
  "id": "learner-replica-table-test",
  "start_key": "7480000000000000ff2d5f720000000000fa",
  "end_key": "7480000000000000ff2e00000000000000f8",
  "role": "learner",
  "count": 2,
  "label_constraints": [
    {"key": "engine", "op": "in", "values": ["tiflash"]}
  ],
  "location_labels": ["host"]
}

Due to the <engine, mpp> label, there is no region that will be moved to the ReadNode.

Hi @JmPotato , I know that the placement rules can make PD add learner peers within a key range to TiFlash node with label <engine, tiflash> and don't add those peers to <engine, mpp>.
But I'm wondering if we start a TiFlash with label <engine, mpp>. How can we prevent PD from treating that TiFlash ReadNode as "TiKV node" and add normal follower peers to that TiFlash ReadNode?