gcs: compatible with old gcs bug #677

3pointer · 2021-01-05T09:08:54Z

What problem does this PR solve?

solve #675

What is changed and how it works?

for case 1: add a check before Read & Write meta file, decide which prefix to use.
for case 2: check whether sst file is in Prefix(after #647) or in PrefixSlash(before #647), adjust proper prefix for TiKV request.

Check List

Tests

Integration test

Related changes

Need to cherry-pick to the release branch
Need to update the documentation

Release Note

Fix the issue that br 4.0.9 cannot restore br 4.0.8 gcs backup files.

kennytm

This PR will also affect all non-BR users of this package. can you do we fix outside gcsStorage? Especially not do the request inside objectName since it affects all GCS calls not just "reading backupmeta"

3pointer · 2021-01-05T10:22:07Z

This PR will also affect all non-BR users of this package. can you do we fix outside gcsStorage? Especially not do the request inside objectName since it affects all GCS calls not just "reading backupmeta"

You are right. Let me think

overvenus

Please move compatibility code to a new package compat.

pkg/task/common.go

3pointer · 2021-01-08T11:50:58Z

/run-integration-tests

3pointer · 2021-01-08T12:01:11Z

/run-integration-tests

3pointer · 2021-01-08T12:09:22Z

/run-integration-tests

overvenus · 2021-01-08T14:56:32Z

LGTM

kennytm

rest LGTM

kennytm · 2021-01-09T13:38:59Z

go.mod1

@@ -38,3 +38,8 @@ require (
 	google.golang.org/grpc v1.27.1
 	sourcegraph.com/sourcegraph/appdash v0.0.0-20190731080439-ebfcffb1b5c0
 )
+
+replace (
+	cloud.google.com/go/storage => github.com/3pointer/google-cloud-go/storage v1.6.1-0.20210108125931-b59bfa0720b2


(googleapis/google-cloud-go#3509 for future reference)

go.mod1

pkg/storage/gcs.go

pkg/task/common.go

tests/br_gcs/oauth.py

Co-authored-by: kennytm <kennytm@gmail.com>

pkg/task/common.go

kennytm

LGTM

3pointer · 2021-01-11T09:10:34Z

/run-integration-test

3pointer · 2021-01-11T09:11:20Z

/run-all-tests

3pointer · 2021-01-11T09:59:50Z

/run-cherry-picker

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

ti-srebot · 2021-01-11T10:01:42Z

cherry pick to release-4.0 in PR #688

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

ti-srebot · 2021-01-11T10:03:11Z

cherry pick to release-5.0-rc in PR #689

* cherry pick #677 to release-4.0 Signed-off-by: ti-srebot <ti-srebot@pingcap.com> * resolve conflict * fix ci * fix tiflash kill issue Co-authored-by: 3pointer <luancheng@pingcap.com>

* cherry pick #677 to release-5.0-rc Signed-off-by: ti-srebot <ti-srebot@pingcap.com> * fix build Co-authored-by: 3pointer <luancheng@pingcap.com>

3pointer added 2 commits January 5, 2021 17:04

gcs: compatible with old gcs bug

3decf96

fix build

1333e9e

kennytm reviewed Jan 5, 2021

View reviewed changes

address comment

0e3beaf

overvenus reviewed Jan 6, 2021

View reviewed changes

pkg/task/common.go Show resolved Hide resolved

overvenus added component/external-storage type/compatibility type/bug-fix needs-cherry-pick-release-4.0 labels Jan 6, 2021

3pointer added 12 commits January 6, 2021 16:44

update go mod

01affc1

update go mod

356d239

fix test

1d8b2d7

add integration test for gcs

f99c53e

fix gcs test

5981352

use forked gcs client go mod

a5abc46

fix test

09fe24c

fix test

c050757

add v4.0.8 br to test backup compatiblity

97b292f

fix ci

5b24aa6

fix ci

b73ffd3

fix ci

44a4e9b

3pointer changed the title ~~WIP: gcs: compatible with old gcs bug~~ gcs: compatible with old gcs bug Jan 8, 2021

3pointer added 3 commits January 8, 2021 19:27

address comment

224f242

Merge branch 'master' into gcs_test

3437b96

solve conflict

1d09ce3

abstract code

09251d8

3pointer added 2 commits January 8, 2021 21:03

go mod: update gcs package

b559a7c

fix build

6a9a3e9

ti-srebot added the status/LGT1 LGTM1 label Jan 8, 2021

kennytm reviewed Jan 9, 2021

View reviewed changes

3pointer and others added 3 commits January 11, 2021 11:15

Apply suggestions from code review

8a8b69b

Co-authored-by: kennytm <kennytm@gmail.com>

addess comment

71d964b

Merge branch 'master' into gcs_test

7d2191c

kennytm reviewed Jan 11, 2021

View reviewed changes

pkg/task/common.go Outdated Show resolved Hide resolved

pkg/task/common.go Outdated Show resolved Hide resolved

address comment

2584619

kennytm reviewed Jan 11, 2021

View reviewed changes

ti-srebot added status/LGT2 LGTM2 and removed status/LGT1 LGTM1 labels Jan 11, 2021

ti-srebot approved these changes Jan 11, 2021

View reviewed changes

3pointer added the needs-cherry-pick-5.0-rc label Jan 11, 2021

fix ci

64e44c7

Merge branch 'master' into gcs_test

e600475

3pointer merged commit 3de76b2 into pingcap:master Jan 11, 2021

ti-srebot pushed a commit to ti-srebot/br that referenced this pull request Jan 11, 2021

cherry pick pingcap#677 to release-4.0

108477e

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

ti-srebot mentioned this pull request Jan 11, 2021

gcs: compatible with old gcs bug (#677) #688

Merged

ti-srebot pushed a commit to ti-srebot/br that referenced this pull request Jan 11, 2021

cherry pick pingcap#677 to release-5.0-rc

df55dff

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

ti-srebot mentioned this pull request Jan 11, 2021

gcs: compatible with old gcs bug (#677) #689

Merged

kennytm pushed a commit that referenced this pull request Jan 11, 2021

gcs: compatible with old gcs bug (#677) (#689)

9adc69b

* cherry pick #677 to release-5.0-rc Signed-off-by: ti-srebot <ti-srebot@pingcap.com> * fix build Co-authored-by: 3pointer <luancheng@pingcap.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gcs: compatible with old gcs bug #677

gcs: compatible with old gcs bug #677

3pointer commented Jan 5, 2021 •

edited

Loading

kennytm left a comment

3pointer commented Jan 5, 2021

overvenus left a comment

3pointer commented Jan 8, 2021

3pointer commented Jan 8, 2021

3pointer commented Jan 8, 2021

overvenus commented Jan 8, 2021

kennytm left a comment

kennytm Jan 9, 2021

kennytm left a comment

3pointer commented Jan 11, 2021

3pointer commented Jan 11, 2021

3pointer commented Jan 11, 2021

ti-srebot commented Jan 11, 2021

ti-srebot commented Jan 11, 2021

gcs: compatible with old gcs bug #677

gcs: compatible with old gcs bug #677

Conversation

3pointer commented Jan 5, 2021 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release Note

kennytm left a comment

Choose a reason for hiding this comment

3pointer commented Jan 5, 2021

overvenus left a comment

Choose a reason for hiding this comment

3pointer commented Jan 8, 2021

3pointer commented Jan 8, 2021

3pointer commented Jan 8, 2021

overvenus commented Jan 8, 2021

kennytm left a comment

Choose a reason for hiding this comment

kennytm Jan 9, 2021

Choose a reason for hiding this comment

kennytm left a comment

Choose a reason for hiding this comment

3pointer commented Jan 11, 2021

3pointer commented Jan 11, 2021

3pointer commented Jan 11, 2021

ti-srebot commented Jan 11, 2021

ti-srebot commented Jan 11, 2021

3pointer commented Jan 5, 2021 •

edited

Loading