Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

snapshot enhancement #2350

Merged
merged 1 commit into from
Nov 14, 2023
Merged

snapshot enhancement #2350

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 23 additions & 20 deletions docs-2.0-en/backup-and-restore/3.manage-snapshot.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,31 +14,15 @@ If authentication is enabled, only the GOD role user can use the snapshot featur

* NebulaGraph cannot automatically delete the invalid files created by a failed snapshot task. You have to manually delete them by using [`DROP SNAPSHOT`](#delete_snapshots).

* Customizing the storage path for snapshots is not supported for now. The default path is `/usr/local/nebula/data`.

## Snapshot form and path

NebulaGraph snapshots are stored in the form of directories with names like `SNAPSHOT_2021_03_09_08_43_12`. The suffix `2021_03_09_08_43_12` is generated automatically based on the creation time (UTC).

When a snapshot is created, snapshot directories will be automatically created in the `checkpoints` directory on the leader Meta server and each Storage server.

To fast locate the path where the snapshots are stored, you can use the Linux command `find`. For example:

```bash
$ find |grep 'SNAPSHOT_2021_03_09_08_43_12'
./data/meta2/nebula/0/checkpoints/SNAPSHOT_2021_03_09_08_43_12
./data/meta2/nebula/0/checkpoints/SNAPSHOT_2021_03_09_08_43_12/data
./data/meta2/nebula/0/checkpoints/SNAPSHOT_2021_03_09_08_43_12/data/000081.sst
...
```
* Customizing the storage path for snapshots is not supported for now.

## Create snapshots

Run `CREATE SNAPSHOT` to create a snapshot for all the graph spaces based on the current time for NebulaGraph. Creating a snapshot for a specific graph space is not supported yet.

!!! note

If the creation fails, [delete the snapshot](#delete_snapshots) and try again.
If the creation fails, refer to the later section to delete the corrupted snapshot and then recreate the snapshot.

```ngql
nebula> CREATE SNAPSHOT;
Expand Down Expand Up @@ -66,6 +50,21 @@ The parameters in the return information are described as follows.
| `Status` | The status of the snapshot. `VALID` indicates that the creation succeeded, while `INVALID` indicates that it failed. |
| `Hosts` | IP addresses and ports of all Storage servers at the time the snapshot was created. |

### Snapshot path

Snapshots are stored in the path specified by the `data_path` parameter in the Meta and Storage configuration files. When a snapshot is created, the `checkpoints` directory is checked in the datastore path of the leader Meta service and all Storage services for the existence, and if it is not there, it is automatically created. The newly created snapshot is stored as a subdirectory within the `checkpoints` directory. For example, `SNAPSHOT_2021_03_09_08_43_12`. The suffix `2021_03_09_08_43_12` is generated automatically based on the creation time (UTC).

To fast locate the path where the snapshots are stored, you can use the Linux command `find` in the datastore path. For example:

```bash
$ cd /usr/local/nebula-graph-ent-{{nebula.release}}/data
$ find |grep 'SNAPSHOT_2021_03_09_08_43_12'
./data/meta2/nebula/0/checkpoints/SNAPSHOT_2021_03_09_08_43_12
./data/meta2/nebula/0/checkpoints/SNAPSHOT_2021_03_09_08_43_12/data
./data/meta2/nebula/0/checkpoints/SNAPSHOT_2021_03_09_08_43_12/data/000081.sst
...
```

## Delete snapshots

To delete a snapshot with the given name, run `DROP SNAPSHOT`.
Expand All @@ -86,6 +85,10 @@ nebula> SHOW SNAPSHOTS;
+--------------------------------+---------+------------------+
```

!!! note

Deleting the only snapshot within the `checkpoints` directory also deletes the `checkpoints` directory.

## Restore data with snapshots

!!! warning
Expand All @@ -94,7 +97,7 @@ nebula> SHOW SNAPSHOTS;

Currently, there is no command to restore data with snapshots. You need to manually copy the snapshot file to the corresponding folder, or you can make it by using a shell script. The logic implements as follows:

1. After the snapshot is created, the `checkpoints` directory is generated in the installation directory of the leader Meta server and all Storage servers, and saves the created snapshot. Taking this topic as an example, when there are two graph spaces, the snapshots created are saved in `/usr/local/nebula/data/meta/nebula/0/checkpoints`, `/usr/local/nebula/data/storage/ nebula/3/checkpoints` and `/usr/local/nebula/data/storage/nebula/4/checkpoints`.
1. After the snapshot is created, the `checkpoints` directory is generated in the installation directory of the leader Meta service and all Storage services, and saves the created snapshot. Taking this topic as an example, when there are two graph spaces, the snapshots created are saved in `/usr/local/nebula/data/meta/nebula/0/checkpoints`, `/usr/local/nebula/data/storage/ nebula/3/checkpoints` and `/usr/local/nebula/data/storage/nebula/4/checkpoints`.

```bash
$ ls /usr/local/nebula/data/meta/nebula/0/checkpoints/
Expand All @@ -109,7 +112,7 @@ Currently, there is no command to restore data with snapshots. You need to manua

!!! warning

The data and wal directories of all Meta servers should be overwritten at the same time. Otherwise, the new leader Meta server will use the latest Meta data after a cluster is restarted.
The data and wal directories of all Meta services should be overwritten at the same time. Otherwise, the new leader Meta service will use the latest Meta data after a cluster is restarted.

<!-- TODO

Expand Down
44 changes: 24 additions & 20 deletions docs-2.0-zh/backup-and-restore/3.manage-snapshot.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,31 +14,15 @@

- 不支持自动回收创建失败的快照垃圾文件，需要手动删除。

- 不支持指定快照保存路径，默认路径为`/usr/local/nebula/data`。

## 快照路径

{{nebula.name}}创建的快照以目录的形式存储，例如`SNAPSHOT_2021_03_09_08_43_12`，后缀`2021_03_09_08_43_12`根据创建时间（UTC）自动生成。

创建快照时，快照目录会自动在 leader Meta 服务器和所有 Storage 服务器的目录`checkpoints`内创建。

为了快速定位快照所在路径，可以使用 Linux 命令`find`。例如：

```bash
$ find |grep 'SNAPSHOT_2021_03_09_08_43_12'
./data/meta2/nebula/0/checkpoints/SNAPSHOT_2021_03_09_08_43_12
./data/meta2/nebula/0/checkpoints/SNAPSHOT_2021_03_09_08_43_12/data
./data/meta2/nebula/0/checkpoints/SNAPSHOT_2021_03_09_08_43_12/data/000081.sst
...
```
- 不支持修改快照保存路径。

## 创建快照

命令`CREATE SNAPSHOT`可以创建集群当前时间点的快照。只支持创建所有图空间的快照，不支持创建指定图空间的快照。

!!! note

如果快照创建失败，请[删除快照](#_7)重新创建。
如果快照创建失败，请参考后文删除损坏的快照，然后重新创建快照。

```ngql
nebula> CREATE SNAPSHOT;
Expand Down Expand Up @@ -66,6 +50,21 @@ nebula> SHOW SNAPSHOTS;
|`Status`|快照状态。`VALID`表示快照有效，`INVALID`表示快照无效。|
|`Hosts`|创建快照时所有 Storage 服务器的 IP 地址和端口。|

### 快照路径

快照保存在 Meta 和 Storage 配置文件中`data_path`参数指定的路径中。创建快照时，在 leader Meta 服务和所有 Storage 服务的数据存储路径中会检查是否有`checkpoints`目录，如果没有会自动创建。新建的快照以子目录的形式储存在`checkpoints`目录内。例如`SNAPSHOT_2021_03_09_08_43_12`，后缀`2021_03_09_08_43_12`根据创建时间（UTC）自动生成。

为了快速定位快照所在路径，可以在数据存储路径内使用 Linux 命令`find`。例如：

```bash
$ cd /usr/local/yueshu-graph-{{nebula.release}}/data
$ find |grep 'SNAPSHOT_2021_03_09_08_43_12'
./data/meta2/nebula/0/checkpoints/SNAPSHOT_2021_03_09_08_43_12
./data/meta2/nebula/0/checkpoints/SNAPSHOT_2021_03_09_08_43_12/data
./data/meta2/nebula/0/checkpoints/SNAPSHOT_2021_03_09_08_43_12/data/000081.sst
...
```

## 删除快照

命令`DROP SNAPSHOT`可以删除指定的快照，语法为：
Expand All @@ -85,6 +84,11 @@ nebula> SHOW SNAPSHOTS;
| "SNAPSHOT_2021_03_09_09_10_52" | "VALID" | "127.0.0.1:9779" |
+--------------------------------+---------+------------------+
```

!!! note

删除最后一个快照时，会将`checkpoints`目录一起删除。

## 恢复快照

!!! warning
Expand All @@ -93,7 +97,7 @@ nebula> SHOW SNAPSHOTS;

当前未提供恢复快照命令，需要手动拷贝快照文件到对应的文件夹内，也可以通过 shell 脚本进行操作。实现逻辑如下：

1. 创建快照后，会在 leader Meta 服务器和所有 Storage 服务器的安装目录内生成`checkpoints`目录，保存创建的快照。以本文为例，当存在 2 个图空间时，创建的快照分别保存在`/usr/local/nebula/data/meta/nebula/0/checkpoints`、`/usr/local/nebula/data/storage/nebula/3/checkpoints`和`/usr/local/nebula/data/storage/nebula/4/checkpoints`中。
1. 创建快照后，会在 leader Meta 服务和所有 Storage 服务的安装目录内生成`checkpoints`目录，保存创建的快照。以本文为例，当存在 2 个图空间时，创建的快照分别保存在`/usr/local/nebula/data/meta/nebula/0/checkpoints`、`/usr/local/nebula/data/storage/nebula/3/checkpoints`和`/usr/local/nebula/data/storage/nebula/4/checkpoints`中。

```bash
$ ls /usr/local/nebula/data/meta/nebula/0/checkpoints/
Expand All @@ -108,7 +112,7 @@ nebula> SHOW SNAPSHOTS;

!!! warning

需要同时覆盖所有 Meta 节点的 data 和 wal 目录，因为存在重启集群后发生 Meta 重新选举 leader 的情况，如果不覆盖所有 Meta 节点，新的 leader 使用的还是最新的 Meta 数据，导致恢复失败。
需要同时覆盖所有 Meta 服务的 data 和 wal 目录，因为存在重启集群后发生 Meta 服务重新选举 leader 的情况，如果不覆盖所有 Meta 服务，新的 leader 使用的还是最新的 Meta 服务数据，导致恢复失败。

<!-- TODO

Expand Down