Refine backup and restore documentation

charts/tidb-backup/templates/backup-job.yaml

@@ -2,7 +2,7 @@
 apiVersion: batch/v1
 kind: Job
 metadata:
-  name: {{ .Values.clusterName }}-{{ .Values.name }}
+  name: {{ .Values.clusterName }}-{{ tpl .Values.name . }}
   labels:
     app.kubernetes.io/name: {{ template "chart.name" . }}
     app.kubernetes.io/managed-by: {{ .Release.Service }}

charts/tidb-backup/templates/backup-pvc.yaml

@@ -2,12 +2,13 @@
 kind: PersistentVolumeClaim
 apiVersion: v1
 metadata:
-  name: {{ .Values.name }}
+  name: {{ tpl .Values.name . }}
   labels:
     app.kubernetes.io/name: {{ template "chart.name" . }}
     app.kubernetes.io/managed-by: tidb-operator
     app.kubernetes.io/instance: {{ .Release.Name }}
     app.kubernetes.io/component: backup
+    pingcap.com/backup-cluster-name: {{ .Values.clusterName }}
     helm.sh/chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+"  "_" }}
 spec:
   accessModes:

charts/tidb-backup/values.yaml

@@ -7,7 +7,7 @@ clusterName: demo
 
 mode: backup # backup | restore
 # name is the backup name
-name: fullbackup-20190306
+name: fullbackup-{{ date "200601021504" .Release.Time }}
 image:
   pullPolicy: IfNotPresent
   binlog: pingcap/tidb-binlog:v2.1.8

charts/tidb-cluster/templates/scheduled-backup-pvc.yaml

@@ -8,6 +8,7 @@ metadata:
     app.kubernetes.io/managed-by: tidb-operator
     app.kubernetes.io/instance: {{ .Release.Name }}
     app.kubernetes.io/component: scheduled-backup
+    pingcap.com/backup-cluster-name: {{ template "cluster.name" . }}
     helm.sh/chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+"  "_" }}
 spec:
   accessModes:

docs/backup-restore.md

@@ -0,0 +1,109 @@
+# Backup and Restore TiDB Cluster
+
+## Overview
+
+TiDB Operator supports two kinds of backup:
+
+* [Full backup](#full-backup)(scheduled or ad-hoc) via [`mydumper`](https://www.pingcap.com/docs/dev/reference/tools/mydumper/), which helps you logical backup the TiDB cluster.
+* [Incremental backup](#incremental-backup) via [`TiDB-Binlog`](https://www.pingcap.com/docs/dev/reference/tools/tidb-binlog/overview/), which helps you synchronize the data in the TiDB cluster to other database or backup the data at real-time.
+
+Currently, tidb-operator only supports automatic [restore operation](#restore) for full backup taken by `mydumper`. Restore the backup data captured by `TiDB-Binlog` requires human intervention.
+
+## Full backup
+
+Full backup using `mydumper` to take the logical backup of TiDB cluster. The backup job will create a PVC to store backup data.
+
+By default, the backup uses PV to store the backup data. You can also store the backup data to [Google Cloud Storage](https://cloud.google.com/storage/) bucket or [Ceph Object Storage](https://ceph.com/ceph-storage/object-storage/) by changing the configuration. This way the PV temporarily stores backup data before it is placed in object storage. Refer to [TiDB cluster Backup configuration](./references/tidb-backup-configuration.md) for full configuration guide of backup and restore.
+
+You can either setup a scheduled full backup or take a full backup in ad-hoc manner.
+
+### Scheduled full backup
+
+Scheduled full backup is created along side the TiDB cluster, and it runs periodically like the crontab job.
+
+To configure a scheduled full backup, modify the `scheduledBackup` section in the `charts/tidb-cluster/values.yaml` of the tidb cluster:
+
+* Set `scheduledBackup.create` to `true`
+* Set `scheduledBackup.storageClassName` to the PV storage class name used for backup data
+
+> **Note:** You must set the scheduled full backup PV's [reclaim policy](https://kubernetes.io/docs/tasks/administer-cluster/change-pv-reclaim-policy) to `Retain` to keep your backup data safe.
+
+* Configure `scheduledBackup.schedule` in the [Cron](https://en.wikipedia.org/wiki/Cron) format to define the scheduling
+* `scheduledBakcup.user` and `scheduledBackup.password` must be set to the correct user which has the permission to read the database to be backuped.
+
+Then, create a new cluster with the scheduled full backup configured by `helm install`, or enabling scheduled full backup for existing cluster by `helm upgrade`:
+
+```shell
+$ helm upgrade ${RELEASE_NAME} charts/tidb-cluster -f charts/tidb-cluster/values.yaml
+```
+
+### Ad-Hoc full backup
+
+Ad-hoc backup runs to complete for once. This functionality is encapsulated in another helm chart, `charts/tidb-backup`. According to the `mode` in `charts/tidb-backup/values.yaml`, this chart can perform either full backup or restore. We will cover restore operation in the [restore section](#restore) of this document. 
+
+To create an ad-hoc full backup job, modify the `charts/tidb-backup/values.yaml`:
+
+* Set the `clusterName` to the target TiDB cluster name
+* Set `mode` to `backup`
+* Set `storage.className` to the PV storage class name used for backup data
+* Adjust the `storage.size` according to your database size
+
+> **Note:** You must set the ad-hoc full backup PV's [reclaim policy](https://kubernetes.io/docs/tasks/administer-cluster/change-pv-reclaim-policy) to `Retain` to keep your backup data safe.
+
+Create a secret containing the user and password that has the permission to backup the database:
+
+```shell
+$ kubectl create secret generic backup-secret -n ${namespace} --from-literal=user=<user> --from-literal=password=<password>
+```
+
+Then run the following command to create an ad-hoc backup job:
+
+```shell
+$ helm install charts/tidb-backup --name=<backup-name> --namespace=${namespace}
+```
+
+### View backups
+
+For backups stored in PV, you can view the PVs by the following command:
+
+```shell
+$ kubectl get pvc -n ${namespace} -l app.kubernetes.io/component=backup,pingcap.com/backup-cluster-name=${cluster_name}
+```
+
+If you store your backup data to [Google Cloud Storage](https://cloud.google.com/storage/) or [Ceph Object Storage](https://ceph.com/ceph-storage/object-storage/), you may view the backups by the related GUI or CLI tool.
+
+## Restore
+
+The helm chart `charts/tidb-backup` helps restoring a TiDB cluster using backup data. To perform a restore operation, modify the `charts/tidb-backup/values.yaml`:
+
+* Set the `clusterName` to the target TiDB cluster name
+* Set the `mode` to `restore`
+* Set the `name` to the backup name you want to restore([view backups](#view-backups) helps you view all the backups available). If the backup is stored in `Google Cloud Storage` or `Ceph Object Storage`, you have to configure the corresponding section too(likely, you will continue to use the same configuration you set in the [adhoc full backup](#ad-hoc-full-backup)).
+
+Create a secret containing the user and password that has the permission to restore the database (skip this if you've already created one in the [adhoc full backup](#ad-hoc-full-backup) section):
+
+```shell
+$ kubectl create secret generic backup-secret -n ${namespace} --from-literal=user=<user> --from-literal=password=<password>
+```
+
+Then, restore the backup:
+```shell
+$ helm install charts/tidb-backup --namespace=${namespace}
+```
+
+## Incremental backup
+
+Incremental backup leverage the [`TiDB-Binlog`](https://www.pingcap.com/docs/dev/reference/tools/tidb-binlog/overview/) tool to collect binlog data from TiDB and provide real-time backup and synchronization to downstream platforms.
+
+Incremental backup is disabled in the TiDB cluster by default. To create a TiDB cluster with incremental backup enabled or enable incremental backup in existing TiDB cluster, you have to modify the `charts/tidb-cluster/values.yaml`:
+
+* Set `binlog.pump.create` to `true`
+* Set `binlog.drainer.create` to `true`
+* Set `binlog.pump.storageClassName` and `binlog.drainer.storageClassName` to a proper `storageClass` available in your kubernetes cluster
+* Set `binlog.drainer.destDBType` to your desired downstream, explained in detail below
+
+There's three types of downstream available for incremental backup:
+
+* PersistenceVolume: default downstream. You may consider configuring a large PersistenceVolume for `drainer` (the `binlog.drainer.storage` variable) in this case
+* MySQL compatible database: enable by setting the `binlog.drainer.destDBType` to `mysql`. You have to configure the target address and credential in the `binlog.drainer.mysql` section too.
+* Kafka: enable by setting the `binlog.drainer.destDBType` to `kafka`. You have to configure the zookeeper address and kafka address in the `binlog.drainer.kafka` section too.

docs/operation-guide.md

@@ -222,68 +222,8 @@ To retrieve logs from multiple pods, [`stern`](https://github.com/wercker/stern)
 $ stern -n ${namespace} tidb -c slowlog
 ```
 
-## Backup
+## Backup and Restore
 
-Currently, TiDB Operator supports two kinds of backup: incremental backup via binlog and full backup(scheduled or ad-hoc) via [Mydumper](https://github.com/maxbube/mydumper).
+TiDB Operator provides highly automated backup and recovery operations for TiDB cluster. You can easily take full backup or setup incremental backup of TiDB cluster, and restore the TiDB cluster when the cluster fails.
 
-### Incremental backup
-
-To enable incremental backup, set `binlog.pump.create` and `binlog.drainer.create` to `true`. By default the incremental backup data is stored in protobuffer format in a PV. You can change `binlog.drainer.destDBType` from `pb` to `mysql` or `kafka` and configure the corresponding downstream.
-
-### Full backup
-
-Currently, full backup requires a PersistentVolume. The backup job will create a PVC to store backup data.
-
-By default, the backup uses PV to store the backup data.
-> **Note:** You must set the ad-hoc full backup PV's [reclaim policy](https://kubernetes.io/docs/tasks/administer-cluster/change-pv-reclaim-policy) to `Retain` to keep your backup data safe.
-
-You can also store the backup data to [Google Cloud Storage](https://cloud.google.com/storage/) bucket or [Ceph object storage](https://ceph.com/ceph-storage/object-storage/) by configuring the corresponding section in `values.yaml`. This way the PV temporarily stores backup data before it is placed in object storage.
-
-The comments in `values.yaml` is self-explanatory for both GCP backup and Ceph backup.
-
-### Scheduled full backup
-
-Scheduled full backup can be ran periodically just like crontab job.
-
-To create a scheduled full backup job, modify `scheduledBackup` section in `values.yaml` file.
-
-* `create` must be set to `true`
-* Set `storageClassName` to the PV storage class name used for backup data
-* `schedule` takes the [Cron](https://en.wikipedia.org/wiki/Cron) format
-* `user` and `password` must be set to the correct user which has the permission to read the database to be backuped.
-
-> **Note:** You must set the scheduled full backup PV's [reclaim policy](https://kubernetes.io/docs/tasks/administer-cluster/change-pv-reclaim-policy) to `Retain` to keep your backup data safe.
-
-
-### Ad-Hoc full backup
-
-> **Note:** The rest of the document will use `values.yaml` to reference `charts/tidb-backup/values.yaml`
-
-Ad-Hoc full backup can be done once just like job.
-
-To create an ad-hoc full backup job, modify `backup` section in `values.yaml` file.
-
-* `mode` must be set to `backup`
-* Set `storage.className` to the PV storage class name used for backup data
-
-Create a secret containing the user and password that has the permission to backup the database:
-
-```shell
-$ kubectl create secret generic backup-secret -n ${namespace} --from-literal=user=<user> --from-literal=password=<password>
-```
-
-Then run the following command to create an ad-hoc backup job:
-
-```shell
-$ helm install charts/tidb-backup --name=<backup-name> --namespace=${namespace}
-```
-
-## Restore
-
-Restore is similar to backup. See the `values.yaml` file for details.
-
-Modified the variables in `values.yaml` and then create restore job using the following command:
-
-```shell
-$ helm install charts/tidb-backup --name=<backup-name> --namespace=${namespace}
-```
+For detail operation guides of backup and restore, please refer to [Backup and Restore TiDB Cluster](./backup-restore.md).

docs/references/tidb-backup-configuration.md

@@ -0,0 +1,76 @@
+# TiDB Backup Configuration Reference
+
+`TiDB-Backup` is a helm chart designed for TiDB cluster backup and restore via the [`mydumper`](https://www.pingcap.com/docs/dev/reference/tools/mydumper/) and [`loader`](https://www.pingcap.com/docs-cn/tools/loader/). This documentation will explain the configurations of `TiDB-Backup`, you may refer to [Restore and Backup TiDB cluster](#tidb-backup-configuration-reference) for user guide with example.
+
+## Configurations
+
+### `mode`
+
+- To choose the operation, backup or restore, required
+- Default: "backup"
+
+### `clusterName`
+
+- The TiDB cluster name that should backup from or restore to, required
+- Default: "demo"
+
+### `name`
+
+- The backup name
+- Default: "fullbackup-${date}", date is the start time of backup, accurate to minute
+
+### `secretName`
+
+- The name of the secret which stores user and password used for backup/restore
+- Default: "backup-secret"
+- You can create the secret by `kubectl create secret generic backup-secret --from-literal=user=root --from-literal=password=<password>`
+
+### `storage.className`
+
+- The storageClass used to store the backup data
+- Default: "local-storage"
+
+### `storage.size`
+
+- The storage size of PersistenceVolume
+- Default: "100Gi"
+
+### `backupOptions`
+
+- The options that passed to [`mydumper`](https://github.com/maxbube/mydumper/blob/master/docs/mydumper_usage.rst#options)
+- Default: "--chunk-filesize=100"
+
+### `restoreOptions`
+
+- The options that passed to [`loader`](https://www.pingcap.com/docs-cn/tools/loader/)
+- Default: "-t 16"
+
+### `gcp.bucket`
+
+- The GCP bucket name to store backup data
+
+> **Note**: Once you set any variables under `gcp` section, the backup data will be uploaded to Google Cloud Storage, namely, you have to keep the configuration intact.
+
+### `gcp.secretName`
+
+- The name of the secret which stores the gcp service account credentials json file
+- You can create the secret by `kubectl create secret generic gcp-backup-secret --from-file=./credentials.json`. To download credentials json, refer to [Google Cloud Documentation](https://cloud.google.com/docs/authentication/production#obtaining_and_providing_service_account_credentials_manually)
+
+### `ceph.endpoint`
+
+- The endpoint of ceph object storage
+
+> **Note**: Once you set any variables under `ceph` section, the backup data will be uploaded to ceph object storage, namely, you have to keep the configuration intact.
+
+### `ceph.bucket`
+
+- The bucket name of ceph object storage
+
+### `ceph.secretName`
+
+- The name of the secret which stores ceph object store access key and secret key
+- You can create the secret by:
+
+```shell
+$ kubectl create secret generic ceph-backup-secret --from-literal=access_key=<access-key> --from-literal=secret_key=<secret-key>
+```