restore database from cloud #160

xiaojingchen · 2018-10-31T12:23:13Z

this PR add restore job, the feature can restore data to a TiDB cluster, the premise is that the data is saved in the cloud (recommended gcs)

* add readiness probes for tidb service

gregwebs · 2018-10-31T13:52:47Z

What's the plan to make this scale?

Download one file first
Start loading the first file while downloading the second
Delete the first file once it is loaded.

weekface · 2018-11-01T09:53:20Z

charts/tidb-cluster/templates/restore-job.yaml

+        secret:
+          secretName: {{ .Values.clusterName }}-restore-secret
+    {{- end }}
+{{- end }}


add new line.

weekface · 2018-11-01T09:53:34Z

charts/tidb-cluster/templates/restore-pvc.yaml

+    requests:
+      storage: {{ .Values.restore.storage }}
+  storageClassName: {{ .Values.restore.storageClassName }}
+{{- end }}


weekface · 2018-11-01T09:55:28Z

charts/tidb-cluster/templates/restore-secret.yaml

+    app.kubernetes.io/name: {{ template "chart.name" . }}
+    app.kubernetes.io/managed-by: {{ .Release.Service }}
+    app.kubernetes.io/instance: {{ .Values.clusterName }}
+    app.kubernetes.io/component: restore-secret


Suggested change

app.kubernetes.io/component: restore-secret

app.kubernetes.io/component: restore

weekface · 2018-11-01T09:56:33Z

charts/tidb-cluster/templates/restore-secret.yaml

+apiVersion: v1
+kind: Secret
+metadata:
+  name: {{ .Values.clusterName }}-restore-secret


Suggested change

name: {{ .Values.clusterName }}-restore-secret

name: {{ .Values.clusterName }}-restore

weekface · 2018-11-01T09:57:06Z

charts/tidb-cluster/templates/restore-secret.yaml

+  credentials.json: {{ .Values.restore.gcp.credentialsData }}
+{{- end }}
+  user-password: {{ .Values.restore.password | b64enc | quote }}
+{{- end }}


add new line

weekface · 2018-11-01T09:57:49Z

charts/tidb-cluster/templates/scripts/_start_fullbackup.sh.tpl

@@ -5,7 +5,7 @@ host=`echo {{ .Values.clusterName }}_TIDB_SERVICE_HOST | tr '[a-z]' '[A-Z]'`
 mkdir -p /data/${dirname}/
 cp /savepoint-dir/savepoint /data/${dirname}/

-mydumper \
+./mydumper \


Why change mydumper to ./mydumper?

new version image: pingcap/tidb-cloud-backup:latest will base pingcap/tidb-enterprise-tools:latest , the mydumper and loader's path change.
look at:
tennix/tidb-cloud-backup#1

If so, please use an absolute path.

weekface · 2018-11-01T09:58:01Z

charts/tidb-cluster/templates/scripts/_start_restore.sh.tpl

+  -p ${TIDB_PASSWORD} \
+  -P 4000 \
+  {{ .Values.restore.options }}
+{{- end }}


add new line

weekface · 2018-11-01T10:00:32Z

charts/tidb-cluster/templates/scripts/_start_restore.sh.tpl

+set -euo pipefail
+
+dirname=`date +%Y-%m-%dT%H%M%S`-${MY_POD_NAME}
+mkdir -p /data/${dirname}/


difine /data/${dirname}/ as a var.

weekface · 2018-11-01T10:01:07Z

charts/tidb-cluster/templates/scripts/_start_restore.sh.tpl

+  --destDir=/data/${dirname}
+
+dataDir=/data/${dirname}/{{ .Values.restore.gcp.srcDir }}
+./loader \


Suggested change

./loader \

loader \

?

tennix · 2018-11-01T12:17:36Z

@gregwebs I think it's a long-term plan to make loader support cloud natively.
/cc @GregoryIan

xiaojingchen · 2018-11-02T06:00:00Z

@gregwebs the plan is:

download all the data files from gcs to pv first
the loader restore the datas to tidb

if any step failed then the job is failed and quit

xiaojingchen · 2018-11-02T07:21:21Z

/run-e2e-tests

weekface

LGTM

tennix

LGTM

gregwebs · 2018-11-06T15:09:55Z

charts/tidb-cluster/templates/restore-job.yaml

+            secretKeyRef:
+              name: {{ .Values.clusterName }}-restore-secret
+              key: user-password
+        restartPolicy: OnFailure


This is here a second time.

gregwebs · 2018-11-06T22:48:38Z

charts/tidb-cluster/templates/scripts/_start_restore.sh.tpl

+host=`echo {{ .Values.clusterName }}_TIDB_SERVICE_HOST | tr '[a-z]' '[A-Z]'`
+
+{{- if .Values.restore.gcp }}
+downloader \


This fails for me. /bin/sh: line 7: downloader: command not found

The latest image is just pushed to DockerHub.

…/tidb-operator into restore_base_backup

xiaojingchen · 2018-11-07T06:17:18Z

/run-e2e-tests

xiaojingchen · 2018-11-07T08:05:38Z

/run-e2e-tests

weekface · 2018-11-07T09:06:34Z

/run-e2e-tests

xiaojingchen · 2018-11-07T10:04:02Z

/run-e2e-tests

weekface

LGTM

* TiDB readiness probes (pingcap#147) * add readiness probes for tidb service * fix script

Signed-off-by: liubo02 <liubo02@pingcap.com>

TiDB readiness probes (pingcap#147)

ab2f1c2

* add readiness probes for tidb service

Merge branch 'master' into restore_base_backup

1969b98

xiaojingchen force-pushed the restore_base_backup branch from f164b5b to 1969b98 Compare November 1, 2018 06:24

xiaojingchen added 2 commits November 1, 2018 14:45

Merge branch 'master' into restore_base_backup

453713c

typo

0928a45

xiaojingchen changed the title ~~[WIP] restore database from cloud~~ restore database from cloud Nov 1, 2018

fix script

a7e612d

weekface reviewed Nov 1, 2018

View reviewed changes

tennix self-assigned this Nov 1, 2018

xiaojingchen added 2 commits November 2, 2018 14:01

address comment

ec1c0b3

address comment

2de846a

xiaojingchen and others added 3 commits November 2, 2018 21:49

Merge branch 'master' into restore_base_backup

1398465

address comment

d9d003b

Merge branch 'master' into restore_base_backup

1b6864a

weekface previously approved these changes Nov 6, 2018

View reviewed changes

tennix previously approved these changes Nov 6, 2018

View reviewed changes

gregwebs reviewed Nov 6, 2018

View reviewed changes

xiaojingchen added 3 commits November 7, 2018 11:48

address comment

b0d0450

Merge branch 'master' into restore_base_backup

4874028

Merge branch 'restore_base_backup' of https://github.com/xiaojingchen…

01140c1

…/tidb-operator into restore_base_backup

xiaojingchen dismissed stale reviews from tennix and weekface via 01140c1 November 7, 2018 06:16

weekface approved these changes Nov 7, 2018

View reviewed changes

tennix approved these changes Nov 7, 2018

View reviewed changes

tennix merged commit 744fd54 into pingcap:master Nov 7, 2018

weekface mentioned this pull request Nov 29, 2018

Add cluster restore support #148

Closed

queenliuxx pushed a commit to queenliuxx/tidb-operator that referenced this pull request Dec 19, 2018

restore database from cloud (pingcap#160)

90e2c30

* TiDB readiness probes (pingcap#147) * add readiness probes for tidb service * fix script

fgksgf pushed a commit to fgksgf/tidb-operator that referenced this pull request Dec 23, 2024

feat(doc): add convention.md (pingcap#160)

04eb5e0

Signed-off-by: liubo02 <liubo02@pingcap.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

restore database from cloud #160

restore database from cloud #160

xiaojingchen commented Oct 31, 2018

gregwebs commented Oct 31, 2018

weekface Nov 1, 2018

weekface Nov 1, 2018

weekface Nov 1, 2018

weekface Nov 1, 2018

weekface Nov 1, 2018

weekface Nov 1, 2018

xiaojingchen Nov 2, 2018

tennix Nov 5, 2018

weekface Nov 1, 2018

weekface Nov 1, 2018

weekface Nov 1, 2018

xiaojingchen Nov 2, 2018

tennix commented Nov 1, 2018

xiaojingchen commented Nov 2, 2018

xiaojingchen commented Nov 2, 2018

weekface left a comment

tennix left a comment

gregwebs Nov 6, 2018

gregwebs Nov 6, 2018

tennix Nov 7, 2018

xiaojingchen commented Nov 7, 2018

xiaojingchen commented Nov 7, 2018

weekface commented Nov 7, 2018

xiaojingchen commented Nov 7, 2018

weekface left a comment

	app.kubernetes.io/component: restore-secret
	app.kubernetes.io/component: restore

	name: {{ .Values.clusterName }}-restore-secret
	name: {{ .Values.clusterName }}-restore

restore database from cloud #160

restore database from cloud #160

Conversation

xiaojingchen commented Oct 31, 2018

gregwebs commented Oct 31, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tennix commented Nov 1, 2018

xiaojingchen commented Nov 2, 2018

xiaojingchen commented Nov 2, 2018

weekface left a comment

Choose a reason for hiding this comment

tennix left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xiaojingchen commented Nov 7, 2018

xiaojingchen commented Nov 7, 2018

weekface commented Nov 7, 2018

xiaojingchen commented Nov 7, 2018

weekface left a comment

Choose a reason for hiding this comment