Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SyncTimestamp might be skipped unexpectedly #7138

Closed
JmPotato opened this issue Sep 22, 2023 · 0 comments · Fixed by #7139
Closed

SyncTimestamp might be skipped unexpectedly #7138

JmPotato opened this issue Sep 22, 2023 · 0 comments · Fixed by #7139
Labels
component/tso Timestamp Oracle. severity/critical type/bug The issue is confirmed as a bug.

Comments

@JmPotato
Copy link
Member

Bug Report

#7031 introduced a new path to skip SyncTimestamp in some cases, which might be triggered unexpectedly. For example, if it's the first time that a timestampOracle is created, both last and lastSavedTime could be zero and cause the sync to return early.

pd/pkg/tso/tso.go

Lines 158 to 175 in 301b917

func (t *timestampOracle) SyncTimestamp(leadership *election.Leadership) error {
t.metrics.syncEvent.Inc()
failpoint.Inject("delaySyncTimestamp", func() {
time.Sleep(time.Second)
})
last, err := t.storage.LoadTimestamp(t.tsPath)
if err != nil {
return err
}
lastSavedTime := t.getLastSavedTime()
// If `lastSavedTime` is not zero, it means that the `timestampOracle` has already been initialized
// before, so we could safely skip the sync if `lastSavedTime` is equal to `last`.
if lastSavedTime != typeutil.ZeroTime && typeutil.SubRealTimeByWallClock(lastSavedTime, last) == 0 {
t.metrics.skipSyncEvent.Inc()
return nil
}

@JmPotato JmPotato added type/bug The issue is confirmed as a bug. component/tso Timestamp Oracle. labels Sep 22, 2023
ti-chi-bot bot added a commit that referenced this issue Sep 25, 2023
close #7138

Only skip the sync if the following conditions are met:

1. The timestamp in memory has been initialized.
2. The last saved timestamp in etcd is not zero.
3. The last saved timestamp in memory is not zero.
4. The last saved timestamp in etcd is equal to the last saved timestamp in memory.

Signed-off-by: JmPotato <ghzpotato@gmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component/tso Timestamp Oracle. severity/critical type/bug The issue is confirmed as a bug.
Projects
Development

Successfully merging a pull request may close this issue.

1 participant