Create table with placement policy complains ERROR 1105 (HY000)

[aylei]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1. create a multi-region cluster across us-central1, us-west1 and asia-northeast1 on GCP
2. execute the following sql:

mysql> create placement policy `global` primary_region='us-central1' regions='us-central1,us-west1,asia-northeast1';
Query OK, 0 rows affected (4.58 sec)

mysql> select store_id, address, label from information_schema.tikv_store_status;
+----------+--------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| store_id | address                                                      | label                                                                                                                                                                                                                               |
+----------+--------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|        4 | db-tikv-0.db-tikv-peer.tidb1.svc.us-central1.aylei:20160     | [{"key": "failure-domain.beta.kubernetes.io/zone", "value": "us-central1-b"}, {"key": "kubernetes.io/hostname", "value": "gke-us-central1-b7e4-us-central1-ab8c-665b49c7-v5bs"}, {"key": "region", "value": "us-central1"}]         |
|      169 | db-tikv-0.db-tikv-peer.tidb1.svc.asia-northeast1.aylei:20160 | [{"key": "region", "value": "asia-northeast1"}, {"key": "failure-domain.beta.kubernetes.io/zone", "value": "asia-northeast1-c"}, {"key": "kubernetes.io/hostname", "value": "gke-asia-northeast1--asia-northeast1--409d63a2-nh3v"}] |
|      170 | db-tikv-1.db-tikv-peer.tidb1.svc.asia-northeast1.aylei:20160 | [{"key": "failure-domain.beta.kubernetes.io/zone", "value": "asia-northeast1-b"}, {"key": "kubernetes.io/hostname", "value": "gke-asia-northeast1--asia-northeast1--5daedc21-3973"}, {"key": "region", "value": "asia-northeast1"}] |
|      171 | db-tikv-2.db-tikv-peer.tidb1.svc.asia-northeast1.aylei:20160 | [{"key": "failure-domain.beta.kubernetes.io/zone", "value": "asia-northeast1-a"}, {"key": "kubernetes.io/hostname", "value": "gke-asia-northeast1--asia-northeast1--bd6e926d-wkwf"}, {"key": "region", "value": "asia-northeast1"}] |
|        1 | db-tikv-1.db-tikv-peer.tidb1.svc.us-central1.aylei:20160     | [{"key": "kubernetes.io/hostname", "value": "gke-us-central1-b7e4-us-central1-ab8c-f5bf548b-0dv9"}, {"key": "region", "value": "us-central1"}, {"key": "failure-domain.beta.kubernetes.io/zone", "value": "us-central1-c"}]         |
|        5 | db-tikv-2.db-tikv-peer.tidb1.svc.us-central1.aylei:20160     | [{"key": "failure-domain.beta.kubernetes.io/zone", "value": "us-central1-a"}, {"key": "kubernetes.io/hostname", "value": "gke-us-central1-b7e4-us-central1-ab8c-c13214a7-cbf7"}, {"key": "region", "value": "us-central1"}]         |
|      104 | db-tikv-0.db-tikv-peer.tidb1.svc.us-west1.aylei:20160        | [{"key": "failure-domain.beta.kubernetes.io/zone", "value": "us-west1-a"}, {"key": "kubernetes.io/hostname", "value": "gke-us-west1-2dd8587-us-west1-8c856eb-5c1b05e7-p4bn"}, {"key": "region", "value": "us-west1"}]               |
|      105 | db-tikv-2.db-tikv-peer.tidb1.svc.us-west1.aylei:20160        | [{"key": "region", "value": "us-west1"}, {"key": "failure-domain.beta.kubernetes.io/zone", "value": "us-west1-b"}, {"key": "kubernetes.io/hostname", "value": "gke-us-west1-2dd8587-us-west1-8c856eb-01b36566-0plb"}]               |
|      106 | db-tikv-1.db-tikv-peer.tidb1.svc.us-west1.aylei:20160        | [{"key": "failure-domain.beta.kubernetes.io/zone", "value": "us-west1-c"}, {"key": "kubernetes.io/hostname", "value": "gke-us-west1-2dd8587-us-west1-8c856eb-c63f59b5-d8xm"}, {"key": "region", "value": "us-west1"}]               |
+----------+--------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
9 rows in set (0.64 sec)

mysql> show placement;
+------------------------+-----------------------------------------------------------------------------+
| Target                 | Placement                                                                   |
+------------------------+-----------------------------------------------------------------------------+
| POLICY asia-northeast1 | PRIMARY_REGION="asia-northeast1" REGIONS="asia-northeast1"                  |
| POLICY global          | PRIMARY_REGION="us-central1" REGIONS="us-central1,us-west1,asia-northeast1" |
| POLICY us-central1     | PRIMARY_REGION="us-central1" REGIONS="us-central1"                          |
| POLICY us-west1        | PRIMARY_REGION="us-west1" REGIONS="us-west1"                                |
+------------------------+-----------------------------------------------------------------------------+
4 rows in set (0.46 sec)

mysql> CREATE TABLE promo_code (
    ->     code VARCHAR(64) NOT NULL,
    ->     expiration_time DATETIME NOT NULL,
    ->     UNIQUE (code)
    -> ) PLACEMENT POLICY=`global`;
ERROR 1105 (HY000): failed to notify PD the placement rules: "[PD:placement:ErrBuildRuleList]build rule list failed, needs at least one leader or voter for range {7480000000000000FF4900000000000000F8, 7480000000000000FF4A00000000000000F8}"

mysql> select tidb_version();
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| tidb_version()                                                                                                                                                                                                                                                                                                             |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Release Version: v5.3.0-alpha-1155-g756d0ffc9
Edition: Community
Git Commit Hash: 756d0ffc9f33a330a9137bef8fe2f00115c7d777
Git Branch: master
UTC Build Time: 2021-10-20 07:59:07
GoVersion: go1.16.4
Race Enabled: false
TiKV Min Version: v3.0.0-60965b006877ca7234adaced7890d7b029ed1306
Check Table Before Drop: false |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.24 sec)

2. What did you expect to see? (Required)

Create table succeed with PLACEMENT POLICY=global

3. What did you see instead (Required)

ERROR 1105 (HY000): failed to notify PD the placement rules: "[PD:placement:ErrBuildRuleList]build rule list failed, needs at least one leader or voter for range {7480000000000000FF4900000000000000F8, 7480000000000000FF4A00000000000000F8}"

4. What is your TiDB version? (Required)

Release Version: v5.3.0-alpha-1155-g756d0ffc9
Edition: Community
Git Commit Hash: 756d0ff
Git Branch: master
UTC Build Time: 2021-10-20 07:59:07
GoVersion: go1.16.4
Race Enabled: false
TiKV Min Version: v3.0.0-60965b006877ca7234adaced7890d7b029ed1306
Check Table Before Drop: false

[aylei]

/label sig/sql-infra

[ti-chi-bot]

@aylei: The label(s) sig/sql-infra cannot be applied. These labels are supported: challenge-program, compatibility-breaker, first-time-contributor, contribution, require-LGT3, good first issue, correctness, duplicate, proposal, security, needs-more-info, needs-cherry-pick-3.0, needs-cherry-pick-3.1, needs-cherry-pick-4.0, needs-cherry-pick-5.0, needs-cherry-pick-5.1, needs-cherry-pick-5.2, affects-4.0, affects-5.0, affects-5.1, affects-5.2, backport-4.0.14, backport-4.0.15, backport-5.0.3, backport-5.0.4, backport-5.1.0, backport-5.1.1, backport-5.1.2.

In response to this:

/label sig/sql-infra

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[aylei]

Specifying LEADER_CONSTRAINTS explicitly would work:

mysql> CREATE TABLE promo_code (
    ->     code VARCHAR(64) NOT NULL,
    ->     expiration_time DATETIME NOT NULL,
    ->     UNIQUE (code)
    -> ) LEADER_CONSTRAINTS="[+region=us-central1]"  FOLLOWER_CONSTRAINTS="[+region=us-central1,+region=us-west1,+region=asia-northeast1]";
Query OK, 0 rows affected (5.73 sec)

[AilinKid]

After I deployed a cluster and tracked the PD call stack, I find the main reason is that PD will override the rule in the same bundle with the key made from bundleID + ruleID

Here is some details:

Here will use the bundleID + ruleID as the key in the map

in the reproduced cases above,I found the rules uploaded from TiDB didn't distinguish the ruleID in the same bundle which is done by my previous miss.

So here, I try with my hotfix and test it in the real cluster, this case is nowhere to be found. Maybe this issue is not like the one that wanghe try to solve.

[xhebox]

After I deployed a cluster and tracked the PD call stack, I find the main reason is that PD will override the rule in the same bundle with the key made from bundleID + ruleID

Here is some details:

Here will use the bundleID + ruleID as the key in the map

in the reproduced cases above,I found the rules uploaded from TiDB didn't distinguish the ruleID in the same bundle which is done by my previous miss.

So here, I try with my hotfix and test it in the real cluster, this case is nowhere to be found. Maybe this issue is not like the one that wanghe try to solve.

It is. But now it looks like that there is another failure reason...

[github-actions]

Please check whether the issue should be labeled with 'affects-x.y' or 'fixes-x.y.z', and then remove 'needs-more-info' label.