Skip to content

Latest commit

 

History

History
36 lines (21 loc) · 3.5 KB

2022-08-31-placement-rules-in-SQL-works-with-tables-with-TiFlash-replica.md

File metadata and controls

36 lines (21 loc) · 3.5 KB
Raw

Placement rules in SQL works with tables with TiFlash replica

Introduction

This RFC enables tables to support both setting placement rules and setting tiflash replicas.

Motivation or Background

Currently, there is a conflict between setting tiflash replicas and setting placement rules. For example, if a table has a tiflash replica, it is not supported to set placement rules for this table. Similarly, if a table has set some placement rules, it is not allowed to set tiflash replica for this table. The conflict between tiflash replicas and placement rules has a negative impact on user experience. Thus, this RFC will resolve this conflict, to support setting tiflash replicas and placement rules on tables at the same time.

The conflict between tiflash replicas and placement rules setting is because setting tiflash replicas and placement rules will both generate placement configuration in PD.

  • In the configuration generated by placement rules, group_index = 40, group_override = true.
  • In the configuration generated by tiflash replicas, group_index = 0, group_override = false.

According to the rules in PD, if group_override is true , the configuration will override the configuration of other groups whose group_index is smaller; While if group_override is false, other configurations will not be overridden. Therefore, placement rules will overwrite the setting of tiflash replicas. To avoid this problem, the coexistence of placement rules and tiflash replicas was prohibited in the original development.

Detailed Design

Because placement rules and tiflash replica do not affect each other in their own rules and work independently, we can give a largest value to group_index for the configuration generated by tiflash replicas. We set RuleIndexTiFlash = 120, which is larger than all other placement configurations.

In this way, other placement rules can not override the settings of tiflash replicas. Besides, due to group_override = false in tiflash replicas' configuration, tiflash replicas configuration will not override other placement rules. This allows the tables to support both placement rules and tiflash replicas settings.

Therefore, before setting tiflash replicas(POST /pd/api/v1/config/rule), we call POST /pd/api/v1/config/rule-group to modify the group_index value of the tiflash group as RuleIndexTiFlash.

For the already existing placement configuration of the tiflash replica setting in the old version, we will call POST /pd/api/v1/config/rule-group in the background task of PollTiFlashRoutine to update the group index of the placement configuration generated by the existing tiflash replicas. In this way, we can ensure normal settings for subsequent placement rules.

Investigation & Alternatives

  1. Add setting group_index in the api POST /pd/api/v1/config/rule called by the tiflash replica settings, so that the group_index is RuleIndexTiFlash when the tiflash replica is set.
    • This practice changes the semantics of /pd/api/v1/config/rule, which is inappropriate.
  2. tiflash uses the POST /pd/api/v1/config/placement-rule API to replace the original API.
    • This API will require setting the full number of rules in the group. All the settings of tiflash are in a group, which will lead to a large package of http requests when there are many tiflash tables.
    • Compared with the original code, this change is relatively large, and the introduction of concurrency ddl may cause concurrent coverage problems.