Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

en,zh: add pause deployment docs of tidb cluster #906

Merged
merged 14 commits into from
Dec 29, 2020
1 change: 1 addition & 0 deletions en/TOC.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@
- [Migrate TiDB to Kubernetes](migrate-tidb-to-kubernetes.md)
- [Upgrade TiDB Cluster](upgrade-a-tidb-cluster.md)
- [Upgrade TiDB Operator](upgrade-tidb-operator.md)
- [Pause Sync of TiDB Cluster](pause-sync-of-tidb-cluster.md)
+ Scale TiDB Cluster
- [Manually Scale](scale-a-tidb-cluster.md)
- [Automatically Scale](enable-tidb-cluster-auto-scaling.md)
Expand Down
127 changes: 127 additions & 0 deletions en/pause-sync-of-tidb-cluster.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,127 @@
---
title: Pause Sync of a TiDB Cluster in Kubernetes
summary: Introduce how to pause sync of a TiDB cluster in Kubernetes
---

# Pause Sync of a TiDB Cluster in Kubernetes

This document introduce how to pause sync of a TiDB cluster in Kubernetes with configuration.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
This document introduce how to pause sync of a TiDB cluster in Kubernetes with configuration.
This document introduces how to pause sync of a TiDB cluster in Kubernetes using configuration.

with has multiple meanings. Let's use a more accurate way of expression.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed


## What is sync in TiDB Operator

In TiDB Operator, controller regulates the state of the TiDB cluster in Kubernetes. The controller constantly compares the desired state recorded in the `TidbCluster` object with the actual state of the TiDB cluster. This process is referred to as **sync** generally. For more details, refer to [TiDB Operator Architecture](architecture.md).

## Use scenarios

Here are some cases where you might need to pause sync of a TiDB cluster in Kubernetes.

- Avoiding unexpected rolling update
BinChenn marked this conversation as resolved.
Show resolved Hide resolved

In order to prevent the compatibility of the new version TiDB Operator from affecting the TiDB clusters, you can pause sync of TiDB clusters before updating the TiDB Operator. After updating TiDB Operator, resume sync of TiDB clusters one by one or resume sync of TiDB clusters at a specified time, so as to observe the impact of TiDB Operator rolling update on the cluster.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
In order to prevent the compatibility of the new version TiDB Operator from affecting the TiDB clusters, you can pause sync of TiDB clusters before updating the TiDB Operator. After updating TiDB Operator, resume sync of TiDB clusters one by one or resume sync of TiDB clusters at a specified time, so as to observe the impact of TiDB Operator rolling update on the cluster.
To prevent new versions of TiDB Operator from introducing compatibility issues into the clusters, before updating TiDB Operator, you can pause sync of TiDB clusters. After updating TiDB Operator, you can resume syncing clusters one by one, or specify a time for resume. In this way, you can observe how the rolling update of TiDB Operator would affect the cluster.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed


- Avoid multiple rolling restarts

In some cases, the configuration of a TiDB cluster may be modified several times over a period of time, but you do not want to restart the TiDB cluster many times. In order to avoid multiple rolling restarts, you can pause sync of a TiDB cluster first. During this period, any modification to configuration of the TiDB cluster will not take effect. After the cluster configuration modification is completed, resuming sync of TiDB Cluster. At this time, multiple configuration changes during the sync pausing can be applied in one time rolling restart.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
In some cases, the configuration of a TiDB cluster may be modified several times over a period of time, but you do not want to restart the TiDB cluster many times. In order to avoid multiple rolling restarts, you can pause sync of a TiDB cluster first. During this period, any modification to configuration of the TiDB cluster will not take effect. After the cluster configuration modification is completed, resuming sync of TiDB Cluster. At this time, multiple configuration changes during the sync pausing can be applied in one time rolling restart.
In some cases, you might need to continuously modify the cluster over a period of time, but do not want to restart the TiDB cluster many times. To avoid multiple rolling restarts, you can pause sync of the cluster. During the sync pausing, any change of the configuration does not take effect on the cluster. After you finish the modification, resume sync of the TiDB cluster. All changes can be applied in a single rolling restart.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed


- Maintenance window

In some situations, you can update or restart TiDB cluster only during a maintenance window. When outside the maintenance window, you can pause sync of the TiDB cluster, so that any modification to spec does not take effect. When inside the maintenance window, you can resume sync of the TiDB cluster to allow TiDB cluster rolling update or restart.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
In some situations, you can update or restart TiDB cluster only during a maintenance window. When outside the maintenance window, you can pause sync of the TiDB cluster, so that any modification to spec does not take effect. When inside the maintenance window, you can resume sync of the TiDB cluster to allow TiDB cluster rolling update or restart.
In some situations, you can update or restart the TiDB cluster only during a maintenance window. When outside the maintenance window, you can pause sync of the TiDB cluster, so that any modification to the specs does not take effect. When inside the maintenance window, you can resume sync of the TiDB cluster to allow TiDB cluster to rolling update or restart.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed


## Pause sync

1. Execute the following command to edit configuration of TiDB cluster. `${cluster_name}` represents the name of TiDB cluster, and `${namespace}` refers to the TiDB cluster namespace.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
1. Execute the following command to edit configuration of TiDB cluster. `${cluster_name}` represents the name of TiDB cluster, and `${namespace}` refers to the TiDB cluster namespace.
1. Execute the following command to edit the TiDB cluster configuration. `${cluster_name}` represents the name of the TiDB cluster, and `${namespace}` refers to the TiDB cluster namespace.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed


{{< copyable "shell-regular" >}}

```shell
kubectl edit tc ${cluster_name} -n ${namespace}
```

2. Configure the TidbCluster CR with `spec.paused: true` as following, save changes and exit editor, sync of TiDB cluster's components (PD, TiKV, TiDB, TiFlash, TiCDC,Pump) will be paused.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
2. Configure the TidbCluster CR with `spec.paused: true` as following, save changes and exit editor, sync of TiDB cluster's components (PD, TiKV, TiDB, TiFlash, TiCDC,Pump) will be paused.
2. Configure `spec.paused: true` as follows, and save changes. The sync of TiDB cluster components (PD, TiKV, TiDB, TiFlash, TiCDC, Pump) is then paused.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed


{{< copyable "" >}}

```yaml
apiVersion: pingcap.com/v1alpha1
kind: TidbCluster
metadata:
...
spec:
...
paused: true # Pausing sync of TiDB cluster
pd:
...
tikv:
...
tidb:
...
```

3. Execute the following command to confirm the sync status of a TiDB cluster. `${pod_name}` is the name of tidb-controller-manager Pod, and `${namespace}` is the namespace of TiDB Operator.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
3. Execute the following command to confirm the sync status of a TiDB cluster. `${pod_name}` is the name of tidb-controller-manager Pod, and `${namespace}` is the namespace of TiDB Operator.
3. To confirm the sync status of a TiDB cluster, execute the following command. `${pod_name}` is the name of tidb-controller-manager Pod, and `${namespace}` is the namespace of TiDB Operator.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed


{{< copyable "shell-regular" >}}

```shell
kubectl logs ${pod_name} -n ${namespace} | grep paused
```

The expected output is as follows. The sync of all components in the TiDB cluster is paused.

```
I1207 11:09:59.029949 1 pd_member_manager.go:92] tidb cluster default/basic is paused, skip syncing for pd service
I1207 11:09:59.029977 1 pd_member_manager.go:136] tidb cluster default/basic is paused, skip syncing for pd headless service
I1207 11:09:59.035437 1 pd_member_manager.go:191] tidb cluster default/basic is paused, skip syncing for pd statefulset
I1207 11:09:59.035462 1 tikv_member_manager.go:116] tikv cluster default/basic is paused, skip syncing for tikv service
I1207 11:09:59.036855 1 tikv_member_manager.go:175] tikv cluster default/basic is paused, skip syncing for tikv statefulset
I1207 11:09:59.036886 1 tidb_member_manager.go:132] tidb cluster default/basic is paused, skip syncing for tidb headless service
I1207 11:09:59.036895 1 tidb_member_manager.go:258] tidb cluster default/basic is paused, skip syncing for tidb service
I1207 11:09:59.039358 1 tidb_member_manager.go:188] tidb cluster default/basic is paused, skip syncing for tidb statefulset
```

## Resume sync

If you want to resume the sync of the TiDB cluster, configure the TidbCluster CR with `spec.paused: false`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
If you want to resume the sync of the TiDB cluster, configure the TidbCluster CR with `spec.paused: false`.
To resume the sync of the TiDB cluster, configure `spec.paused: false` in the TidbCluster CR.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed


1. Execute the following command to edit configuration of TiDB cluster. `${cluster_name}` represents the name of TiDB cluster, and `${namespace}` refers to the TiDB cluster namespace.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
1. Execute the following command to edit configuration of TiDB cluster. `${cluster_name}` represents the name of TiDB cluster, and `${namespace}` refers to the TiDB cluster namespace.
1. Execute the following command to edit the TiDB cluster configuration. `${cluster_name}` represents the name of the TiDB cluster, and `${namespace}` refers to the TiDB cluster namespace.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed


{{< copyable "shell-regular" >}}

```shell
kubectl edit tc ${cluster_name} -n ${namespace}
```

2. Configure the TidbCluster CR with `spec.paused: false` as following, save changes and exit editor, sync of TiDB cluster's components (PD, TiKV, TiDB, TiFlash, TiCDC,Pump) will be resumed.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
2. Configure the TidbCluster CR with `spec.paused: false` as following, save changes and exit editor, sync of TiDB cluster's components (PD, TiKV, TiDB, TiFlash, TiCDC,Pump) will be resumed.
2. Configure `spec.paused: false` as follows, and save changes. The sync of TiDB cluster components (PD, TiKV, TiDB, TiFlash, TiCDC, Pump) is then resumed.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed


{{< copyable "" >}}

```yaml
apiVersion: pingcap.com/v1alpha1
kind: TidbCluster
metadata:
...
spec:
...
paused: false # Resuming sync of TiDB cluster
pd:
...
tikv:
...
tidb:
...
```

3. After resuming sync of the TiDB cluster, execute the following command to confirm sync status of the TiDB cluster. `${pod_name}` represents the name of tidb-controller-manager Pod, `${namespace}` represents the namespace of TiDB Operator.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
3. After resuming sync of the TiDB cluster, execute the following command to confirm sync status of the TiDB cluster. `${pod_name}` represents the name of tidb-controller-manager Pod, `${namespace}` represents the namespace of TiDB Operator.
3. To confirm the sync status of a TiDB cluster, execute the following command. `${pod_name}` represents the name of the tidb-controller-manager Pod, and `${namespace}` represents the namespace of TiDB Operator.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed


{{< copyable "shell-regular" >}}

```shell
kubectl logs ${pod_name} -n ${namespace} | grep "Finished syncing TidbCluster"
```

The expected output is as follows. The `finished syncing` timestamp is later than the `pausing` timestamp, which indicates that sync of the TiDB cluster has been resumed.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The expected output is as follows. The `finished syncing` timestamp is later than the `pausing` timestamp, which indicates that sync of the TiDB cluster has been resumed.
The expected output is as follows. The `finished syncing` timestamp is later than the `paused` timestamp, which indicates that sync of the TiDB cluster has been resumed.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed


```
I1207 11:14:59.361353 1 tidb_cluster_controller.go:136] Finished syncing TidbCluster "default/basic" (368.816685ms)
I1207 11:15:28.982910 1 tidb_cluster_controller.go:136] Finished syncing TidbCluster "default/basic" (97.486818ms)
I1207 11:15:29.360446 1 tidb_cluster_controller.go:136] Finished syncing TidbCluster "default/basic" (377.51187ms)
```
1 change: 1 addition & 0 deletions zh/TOC.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@
- [迁移 TiDB 至 Kubernetes](migrate-tidb-to-kubernetes.md)
- [升级 TiDB 集群](upgrade-a-tidb-cluster.md)
- [升级 TiDB Operator](upgrade-tidb-operator.md)
- [暂停 TiDB 集群同步](pause-sync-of-tidb-cluster.md)
+ TiDB 集群伸缩
- [手动扩缩容](scale-a-tidb-cluster.md)
- [自动弹性伸缩](enable-tidb-cluster-auto-scaling.md)
Expand Down
127 changes: 127 additions & 0 deletions zh/pause-sync-of-tidb-cluster.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,127 @@
---
title: 暂停同步 Kubernetes 上的 TiDB 集群
summary: 介绍如何暂停同步 Kubernetes 上的 TiDB 集群
---

# 暂停同步 Kubernetes 上的 TiDB 集群

本文介绍如何通过配置暂停同步 Kubernetes 上的 TiDB 集群。

## 什么是同步

在 TiDB Operator 中,控制器会不断对比 `TidbCluster` 对象中记录的期望状态与 TiDB 集群的实际状态,并调整 Kubernetes 中的资源以驱动 TiDB 集群满足期望状态。这个不断调整的过程通常被称为**同步**。更多细节参见[TiDB Operator架构](architecture.md)。

## 暂停同步的应用场景

以下为一些暂停同步的应用场景。

- 避免意外的滚动升级

为防止 TiDB Operator 新版本的兼容性问题影响集群,升级 TiDB Operator 之前,可以先暂停同步集群。升级 TiDB Operator 之后,逐个恢复同步集群或者在指定时间恢复同步集群,以此来观察 TiDB Operator 版本升级对集群的影响。

- 避免多次滚动重启集群

在某些情况下,一段时间内可能会多次修改 TiDB 集群配置,但是又不想多次滚动重启集群。为了避免多次滚动重启集群,可以先暂停同步集群,在此期间,对 TiDB 集群的任何配置都不会生效。集群配置修改完成后,恢复集群同步,此时暂停同步期间的所有配置修改都能在一次重启过程中被应用。

- 维护时间窗口

在某些情况下,只允许在特定时间窗口内滚动升级或重启 TiDB 集群。因此可以在维护时间窗口之外的时间段暂停 TiDB 集群的同步过程,这样在维护时间窗口之外对 TiDB 集群的任何配置都不会生效;在维护时间窗口内,可以通过恢复 TiDB 集群同步来允许滚动升级或者重启 TiDB 集群。

## 暂停同步 TiDB 集群

1. 使用以下命令修改集群配置,其中 `${cluster_name}` 表示 TiDB 集群名称, `${namespace}` 表示 TiDB 集群所在的 namespace。

{{< copyable "shell-regular" >}}

```shell
kubectl edit tc ${cluster_name} -n ${namespace}
```

2. 在 TidbCluster CR 中以如下方式配置 `spec.paused: true`,保存配置并退出编辑器。TiDB 集群各组件 (PD、TiKV、TiDB、TiFlash、TiCDC、Pump) 的同步过程将会被暂停。

{{< copyable "" >}}

```yaml
apiVersion: pingcap.com/v1alpha1
kind: TidbCluster
metadata:
...
spec:
...
paused: true # 暂停同步
pd:
...
tikv:
...
tidb:
...
```

3. TiDB 集群同步暂停后,可以使用以下命令查看 tidb-controller-manager Pod 日志确认 TiDB 集群同步状态。其中 `${pod_name}` 表示 tidb-controller-manager Pod 的名称,`${namespace}` 表示 TiDB Operator 所在的 namespace。

{{< copyable "shell-regular" >}}

```shell
kubectl logs ${pod_name} -n ${namespace} | grep paused
```

输出类似下方结果则表示 TiDB 集群同步已经暂停。

```
I1207 11:09:59.029949 1 pd_member_manager.go:92] tidb cluster default/basic is paused, skip syncing for pd service
I1207 11:09:59.029977 1 pd_member_manager.go:136] tidb cluster default/basic is paused, skip syncing for pd headless service
I1207 11:09:59.035437 1 pd_member_manager.go:191] tidb cluster default/basic is paused, skip syncing for pd statefulset
I1207 11:09:59.035462 1 tikv_member_manager.go:116] tikv cluster default/basic is paused, skip syncing for tikv service
I1207 11:09:59.036855 1 tikv_member_manager.go:175] tikv cluster default/basic is paused, skip syncing for tikv statefulset
I1207 11:09:59.036886 1 tidb_member_manager.go:132] tidb cluster default/basic is paused, skip syncing for tidb headless service
I1207 11:09:59.036895 1 tidb_member_manager.go:258] tidb cluster default/basic is paused, skip syncing for tidb service
I1207 11:09:59.039358 1 tidb_member_manager.go:188] tidb cluster default/basic is paused, skip syncing for tidb statefulset
```

## 恢复同步 TiDB 集群

如果想要恢复 TiDB 集群的同步,可以在 TidbCluster CR 中配置 `spec.paused: false`,恢复同步 TiDB 集群。

1. 使用以下命令修改集群配置,其中 `${cluster_name}` 表示 TiDB 集群名称, `${namespace}` 表示 TiDB 集群所在的 namespace。

{{< copyable "shell-regular" >}}

```shell
kubectl edit tc ${cluster_name} -n ${namespace}
```

2. 在 TidbCluster CR 中以如下方式配置 `spec.paused: false`,保存配置并退出编辑器。TiDB 集群各组件 (PD、TiKV、TiDB、TiFlash、TiCDC、Pump) 的同步过程将会被恢复。

{{< copyable "" >}}

```yaml
apiVersion: pingcap.com/v1alpha1
kind: TidbCluster
metadata:
...
spec:
...
paused: false # 恢复同步
pd:
...
tikv:
...
tidb:
...
```

3. 恢复 TiDB 集群同步后,可以使用以下命令查看 tidb-controller-manager Pod 日志确认 TiDB 集群同步状态。其中 `${pod_name}` 表示 tidb-controller-manager Pod 的名称,`${namespace}` 表示 TiDB Operator 所在的 namespace。

{{< copyable "shell-regular" >}}

```shell
kubectl logs ${pod_name} -n ${namespace} | grep "Finished syncing TidbCluster"
```

输出类似下方结果,可以看到同步成功时间戳大于暂停同步日志中显示的时间戳,表示 TiDB 集群同步已经被恢复。

```
I1207 11:14:59.361353 1 tidb_cluster_controller.go:136] Finished syncing TidbCluster "default/basic" (368.816685ms)
I1207 11:15:28.982910 1 tidb_cluster_controller.go:136] Finished syncing TidbCluster "default/basic" (97.486818ms)
I1207 11:15:29.360446 1 tidb_cluster_controller.go:136] Finished syncing TidbCluster "default/basic" (377.51187ms)
```