Skip to content

Commit

Permalink
Support sync bucket in lightning (#1629)
Browse files Browse the repository at this point in the history
* Support sync bucket in lightning

Signed-off-by: Aylei <rayingecho@gmail.com>

* fix nodeSelector is not respected in tidb-lightning chart

Signed-off-by: Aylei <rayingecho@gmail.com>

* Fix nodeSelector indention

Signed-off-by: Aylei <rayingecho@gmail.com>
  • Loading branch information
aylei authored Mar 16, 2020
1 parent 784a9bc commit 868612d
Show file tree
Hide file tree
Showing 4 changed files with 21 additions and 0 deletions.
4 changes: 4 additions & 0 deletions charts/tidb-lightning/templates/job.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,10 @@ spec:
{{- if .Values.affinity }}
affinity:
{{ toYaml .Values.affinity | indent 6 }}
{{- end }}
{{- if .Values.nodeSelector }}
nodeSelector:
{{ toYaml .Values.nodeSelector | indent 8 }}
{{- end }}
{{- if .Values.tolerations }}
tolerations:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,8 @@
set -euo pipefail
{{ if .Values.dataSource.remote.directory }}
# rclone sync skip identical files automatically
rclone --config /etc/rclone/rclone.conf sync -P {{ .Values.dataSource.remote.directory}} /data
{{- else -}}
filename=$(basename {{ .Values.dataSource.remote.path }})
if find /data -name metadata | egrep '.*'; then
echo "data already exist"
Expand All @@ -7,3 +11,4 @@ else
rclone --config /etc/rclone/rclone.conf copy -P {{ .Values.dataSource.remote.path }} /data
cd /data && tar xzvf ${filename}
fi
{{- end -}}
10 changes: 10 additions & 0 deletions charts/tidb-lightning/templates/scripts/_start_lightning.sh.tpl
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,16 @@
data_dir={{ .Values.dataSource.local.hostPath }}
{{- else if .Values.dataSource.adhoc.pvcName -}}
data_dir=/var/lib/tidb-lightning/{{ .Values.dataSource.adhoc.backupName | default .Values.dataSource.adhoc.pvcName }}
{{- else if .Values.dataSource.remote.directory -}}
data_dir=/var/lib/tidb-lightning
if [ -z "$(ls -A ${data_dir})" ]; then
if [ ! -z ${FAIL_FAST} ]; then
exit 1
else
echo "No files in data dir, please exec into my container to diagnose"
tail -f /dev/null
fi
fi
{{- else -}}
data_dir=$(dirname $(find /var/lib/tidb-lightning -name metadata 2>/dev/null) 2>/dev/null)
if [ -z $data_dir ]; then
Expand Down
2 changes: 2 additions & 0 deletions charts/tidb-lightning/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,8 @@ dataSource:
storage: 100Gi
secretName: cloud-storage-secret
path: s3:bench-data-us/sysbench/sbtest_16_1e7.tar.gz
# Directory support downloading all files in a remote directory, shadow dataSoure.remote.path if present
# directory: s3:bench-data-us
# If rcloneConfig is configured, then `secretName` will be ignored,
# `rcloneConfig` should only be used for the cases where no sensitive
# information need to be configured, e.g. the configuration as below,
Expand Down

0 comments on commit 868612d

Please sign in to comment.