use sync.Once to avoid to send drop cutover sentry table to mysql twice

[MOON-CLJ]

ref: #745 (comment)

this pr is another way to deal with issue: #737

use mutex to skip the second drop table。

@shlomi-noach pls review, thx in advance。

[shlomi-noach]

What we actually want to do is to protect two different pieces of code; one within the applier transaction, the other in this DropAtomicCutOverSentryTableIfExists() function. And we don't want to serialize them, but instead only run them once. I suggest using a Once.

[MOON-CLJ]

@shlomi-noach

i am not sure how to use https://golang.org/pkg/sync/#Once to ensure the following three places which cause drop table?

1, CreateAtomicCutOverSentryTable

gh-ost/go/logic/applier.go

Line 827 in 49262c5

if err := this.CreateAtomicCutOverSentryTable(); err != nil {

2, drop cut over table

gh-ost/go/logic/applier.go

Line 845 in 49262c5

if _, err := tx.Exec(query); err != nil {

3, defer clean sentry table

gh-ost/go/logic/migrator.go

Line 613 in 49262c5

this.applier.DropAtomicCutOverSentryTableIfExists()

even more, atomicCutOver will retry some times。although will sleep 1s between retry，two call to goroutine AtomicCutOverMagicLock is not serialize，because atomicCutOver don't wait AtomicCutOverMagicLock to be finished。

[shlomi-noach]

@MOON-CLJ the way I see it, there's these two places:

gh-ost/go/logic/applier.go

Lines 862 to 865 in b38814f

	if _, err := tx.Exec(query); err != nil {
	log.Errore(err)
	// We DO NOT return here because we must `UNLOCK TABLES`!
	}

gh-ost/go/logic/migrator.go

Line 613 in b38814f

this.applier.DropAtomicCutOverSentryTableIfExists()

which are racing.
These do not retry per cut-over.

You are absolutely right that the entire cut-over operation can retry. We can create a Once variable per cut-over.

[MOON-CLJ]

@shlomi-noach

"These do not retry per cut-over." yeah, you're right。

sorry, my mistake。i am confused, because in pr #745 i mv defer clean sentry table

gh-ost/go/logic/migrator.go

Line 613 in 49262c5

this.applier.DropAtomicCutOverSentryTableIfExists()

from atomicCutOver to goroutine AtomicCutOverMagicLock。 so there is a race condition between

the last one cut-over

gh-ost/go/logic/applier.go

Line 823 in 1635183

defer this.DropAtomicCutOverSentryTableIfExists()

and the next one cut-over

gh-ost/go/logic/applier.go

Line 819 in 1635183

if err := this.CreateAtomicCutOverSentryTable(); err != nil {

We can create a Once variable per cut-over.

yes, let me try。

[MOON-CLJ]

@shlomi-noach updated

[MOON-CLJ]

@shlomi-noach we have test this pr in hundreds of clusters in our production env for more than 3 months。consider go forward to merge?

[MOON-CLJ]

@shlomi-noach ping

[shlomi-noach]

Sorry for the delay. It takes a context switch to dive into this PR again; I'd need to clear my desk for this. I assume this will be delayed for a bit longer.

[shlomi-noach]

This look good to me.

[timvaillancourt]

This was merged in #888. Thanks @MOON-CLJ!

[shlomi-noach]

Thank you @timvaillancourt 👑

[MOON-CLJ]

Thank you @timvaillancourt