Skip to content

Commit

Permalink
migrate download and ingest into job manager (#1219)
Browse files Browse the repository at this point in the history
  • Loading branch information
cooper-lzy authored Apr 7, 2022
1 parent 6415afa commit e1c5c8c
Show file tree
Hide file tree
Showing 3 changed files with 31 additions and 6 deletions.
25 changes: 25 additions & 0 deletions docs-2.0/3.ngql-guide/4.job-statements.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,31 @@ nebula> SUBMIT JOB STATS;
+------------+
```

## SUBMIT JOB DOWNLOAD/INGEST

The `SUBMIT JOB DOWNLOAD HDFS` and `SUBMIT JOB INGEST` commands are used to import the SST file into Nebula Graph. For detail, see [Import data from SST files](../nebula-exchange/use-exchange/ex-ug-import-from-sst.md)

The `SUBMIT JOB DOWNLOAD HDFS` command will download the SST file on the specified HDFS.

The `SUBMIT JOB INGEST` command will import the downloaded SST file into Nebula Graph.

For example:

```ngql
nebula> SUBMIT JOB DOWNLOAD HDFS "hdfs://192.168.10.100:9000/sst";
+------------+
| New Job Id |
+------------+
| 10 |
+------------+
nebula> SUBMIT JOB INGEST;
+------------+
| New Job Id |
+------------+
| 11 |
+------------+
```

## SHOW JOB

The Meta Service parses a `SUBMIT JOB` request into multiple tasks and assigns them to the nebula-storaged processes. The `SHOW JOB <job_id>` statement shows the information about a specific job and all its tasks in the current graph space.
Expand Down
4 changes: 2 additions & 2 deletions docs-2.0/7.data-security/1.authentication/3.role-list.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,8 +64,8 @@ The privileges of roles and the nGQL statements that each role can use are liste
|Read data|Y|Y|Y|Y|Y|`GO`, `SET`, `PIPE`, `MATCH`, `ASSIGNMENT`, `LOOKUP`, `YIELD`, `ORDER BY`, `FETCH VERTICES`, `Find`, `FETCH EDGES`, `FIND PATH`, `LIMIT`, `GROUP BY`, `RETURN`|
|Write data|Y|Y|Y|Y||`INSERT VERTEX`, `UPDATE VERTEX`, `INSERT EDGE`, `UPDATE EDGE`, `DELETE VERTEX`, `DELETE EDGES`, `DELETE TAG`|
|Show operations|Y|Y|Y|Y|Y|`SHOW`, `CHANGE PASSWORD`|
|Job|Y|Y|Y|Y||`SUBMIT JOB COMPACT`, `SUBMIT JOB FLUSH`, `SUBMIT JOB STATS`, `STOP JOB`, `RECOVER JOB`, `BUILD TAG INDEX`, `BUILD EDGE INDEX`|
|Write space|Y|||||`CREATE SPACE`, `DROP SPACE`, `CREATE SNAPSHOT`, `DROP SNAPSHOT`, `BALANCE`, `ADMIN`, `CONFIG`, `INGEST`, `DOWNLOAD`|
|Job|Y|Y|Y|Y||`SUBMIT JOB COMPACT`, `SUBMIT JOB FLUSH`, `SUBMIT JOB STATS`, `STOP JOB`, `RECOVER JOB`, `BUILD TAG INDEX`, `BUILD EDGE INDEX``INGEST`, `DOWNLOAD`|
|Write space|Y|||||`CREATE SPACE`, `DROP SPACE`, `CREATE SNAPSHOT`, `DROP SNAPSHOT`, `BALANCE`, `ADMIN`, `CONFIG`|
<!-- balance-3.1
|Write space|Y|||||`CREATE SPACE`, `DROP SPACE`, `CREATE SNAPSHOT`, `DROP SNAPSHOT`, `BALANCE ZONE`, `BALANCE LEADER`, `ADMIN`, `CONFIG`, `INGEST`, `DOWNLOAD`|
-->
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -502,26 +502,26 @@ Connect to the Nebula Graph database using the client tool and import the SST fi
2. Run the following command to download the SST file:

```ngql
nebula> DOWNLOAD HDFS "hdfs://<hadoop_address>:<hadoop_port>/<sst_file_path>";
nebula> SUBMIT JOB DOWNLOAD HDFS "hdfs://<hadoop_address>:<hadoop_port>/<sst_file_path>";
```

For example:

```ngql
nebula> DOWNLOAD HDFS "hdfs://*.*.*.*:9000/sst";
nebula> SUBMIT JOB DOWNLOAD HDFS "hdfs://*.*.*.*:9000/sst";
```

2. Run the following command to import the SST file:

```ngql
nebula> INGEST;
nebula> SUBMIT JOB INGEST;
```

!!! note

- To download the SST file again, delete the `download` folder in the space ID in the `data/storage/nebula` directory in the Nebula Graph installation path, and then download the SST file again. If the space has multiple copies, the `download` folder needs to be deleted on all machines where the copies are saved.

- If there is a problem with the import and re-importing is required, re-execute `INGEST;`.
- If there is a problem with the import and re-importing is required, re-execute `SUBMIT JOB INGEST;`.

### Step 6: (optional) Validate data

Expand Down

0 comments on commit e1c5c8c

Please sign in to comment.