non-default keyspace groups use the same timestamp path by mistake #6465
Labels
type/bug
The issue is confirmed as a bug.
Comments
ti-chi-bot bot
pushed a commit
that referenced
this issue
May 16, 2023
…ke" (#6457) close #6453, close #6465 The tso servers are loading keyspace groups asynchronously. Make sure all keyspace groups are available for serving tso requests from corresponding keyspaces by querying IsKeyspaceServing(keyspaceID, the Desired KeyspaceGroupID). if use default keyspace group id in the query, it will always return true as the keyspace will be served by default keyspace group before the keyspace groups are loaded. Signed-off-by: Bin Shi <binshi.bing@gmail.com>
rleungx
pushed a commit
to rleungx/pd
that referenced
this issue
Aug 2, 2023
…ke" (tikv#6457) close tikv#6453, close tikv#6465 The tso servers are loading keyspace groups asynchronously. Make sure all keyspace groups are available for serving tso requests from corresponding keyspaces by querying IsKeyspaceServing(keyspaceID, the Desired KeyspaceGroupID). if use default keyspace group id in the query, it will always return true as the keyspace will be served by default keyspace group before the keyspace groups are loaded. Signed-off-by: Bin Shi <binshi.bing@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Bug Report
What did you do?
Added TestGetMinTS test but it had random failure. The root cause is that non-default keyspace groups use the same timestamp path by mistake, which caused random failure when saving timestamp in etcd due to transaction conflict thus caused keyspace group primary to be reset unexpectedly.
What did you expect to see?
No random test failure in TestGetMinTS.
What did you see instead?
There were random failures due to the root cause described above.
2023-05-15T00:29:19.5492594Z [2023/05/15 00:29:19.548 +00:00] [WARN] [tso.go:289] ["clock offset"] [jet-lag=221.789286ms] [prev-physical=2023/05/15 00:29:19.326 +00:00] [now=2023/05/15 00:29:19.548 +00:00] [update-physical-interval=50ms]
2023-05-15T00:29:19.5600840Z [2023/05/15 00:29:19.559 +00:00] [WARN] [tso.go:289] ["clock offset"] [jet-lag=169.907035ms] [prev-physical=2023/05/15 00:29:19.389 +00:00] [now=2023/05/15 00:29:19.559 +00:00] [update-physical-interval=50ms]
2023-05-15T00:29:19.5651800Z [2023/05/15 00:29:19.564 +00:00] [WARN] [allocator_manager.go:752] ["failed to update allocator's timestamp"] [dc-location=global] [error="[PD:etcd:ErrEtcdTxnConflict]etcd transaction failed, conflicted and rolled back"]
2023-05-15T00:29:19.5668265Z [2023/05/15 00:29:19.566 +00:00] [INFO] [tso.go:375] ["reset the timestamp in memory"]
2023-05-15T00:29:19.5675787Z [2023/05/15 00:29:19.567 +00:00] [INFO] [lease.go:172] ["stop lease keep alive worker"] [purpose="keyspace group primary election"]
2023-05-15T00:29:19.5726002Z [2023/05/15 00:29:19.572 +00:00] [INFO] [leadership.go:211] ["current leadership is deleted"] [leader-key=/ms/7233199744956437750/tso/00001/primary] [purpose="keyspace group primary election"]
2023-05-15T00:29:19.5733101Z [2023/05/15 00:29:19.573 +00:00] [INFO] [leadership.go:211] ["current leadership is deleted"] [leader-key=/ms/7233199744956437750/tso/00001/primary] [purpose="keyspace group primary election"]
2023-05-15T00:29:19.5746233Z [2023/05/15 00:29:19.574 +00:00] [INFO] [global_allocator.go:499] ["the tso primary has changed, try to re-campaign a primary"]
2023-05-15T00:29:19.5750728Z [2023/05/15 00:29:19.574 +00:00] [INFO] [global_allocator.go:507] ["start to campaign the primary"] [campaign-tso-primary-name=http://127.0.0.1:44687-00001]
What version of PD are you using (
pd-server -V)?The text was updated successfully, but these errors were encountered: