From 834e922c8d84f8518216478c494043bf75077615 Mon Sep 17 00:00:00 2001 From: cooper-lzy <78672629+cooper-lzy@users.noreply.github.com> Date: Mon, 22 May 2023 11:33:25 +0800 Subject: [PATCH] Importer 4.0 --- docs-2.0/nebula-importer/use-importer.md | 434 ++++++++++++----------- mkdocs.yml | 6 +- 2 files changed, 236 insertions(+), 204 deletions(-) diff --git a/docs-2.0/nebula-importer/use-importer.md b/docs-2.0/nebula-importer/use-importer.md index 9bbcc9fa392..a20b9d6b3d1 100644 --- a/docs-2.0/nebula-importer/use-importer.md +++ b/docs-2.0/nebula-importer/use-importer.md @@ -1,10 +1,14 @@ # NebulaGraph Importer -NebulaGraph Importer (Importer) is a standalone tool for importing data from CSV files into [NebulaGraph](https://github.com/vesoft-inc/nebula). Importer can read the local CSV file and then import the data into the NebulaGraph database. +NebulaGraph Importer (Importer) is a standalone tool for importing data from CSV files into NebulaGraph. Importer can read and import CSV file data from multiple data sources. -## Scenario +## Features -Importer is used to import the contents of a local CSV file into the NebulaGraph. +- Support multiple data sources, including local, S3, OSS, HDFS, FTP, and SFTP. +- Support importing data from CSV format files. A single file can contain multiple tags, multiple edge types or a mix of both. +- Support connecting to multiple Graph services simultaneously for importing and dynamic load balancing. +- Support reconnect or retry after failure. +- Support displaying statistics in multiple dimensions, including import time, import percentage, etc. Support for printing statistics in Console or logs. ## Advantage @@ -28,17 +32,19 @@ Before using NebulaGraph Importer, make sure: - [Install NebulaGraph by compiling the source code](../4.deployment-and-installation/2.compile-and-install-nebula-graph/1.install-nebula-graph-by-compiling-the-source-code.md) -- Schema is created in NebulaGraph, including space, Tag and Edge type, or set by parameter `clientSettings.postStart.commands`. - -- Golang environment has been deployed on the machine running the Importer. For details, see [Build Go environment](https://github.com/vesoft-inc/nebula-importer/blob/{{importer.branch}}/docs/golang-install-en.md). +- Schema is created in NebulaGraph, including space, Tag and Edge type, or set by parameter `manager.hooks.before.statements`. ## Steps -Configure the YAML file and prepare the CSV file to be imported to use the tool to batch write data to NebulaGraph. +Prepare the CSV file to be imported and configure the YAML file to use the tool to batch write data into NebulaGraph. + +!!! note + + For details about the YAML configuration file, see configuration file description at the end of topic. ### Download binary package and run -1. Download the [binary package](https://github.com/vesoft-inc/nebula-importer/releases/tag/{{importer.tag}}) directly and add execute permission to it. +1. Download the executable [binary package](https://github.com/vesoft-inc/nebula-importer/releases/tag/{{importer.tag}}). 2. Start the service. @@ -48,6 +54,8 @@ Configure the YAML file and prepare the CSV file to be imported to use the tool ### Source code compile and run +Compiling the source code requires deploying a Golang environment. For details, see [Build Go environment](https://github.com/vesoft-inc/nebula-importer/blob/{{importer.branch}}/docs/golang-install-en.md). + 1. Clone repository. ```bash @@ -56,8 +64,7 @@ Configure the YAML file and prepare the CSV file to be imported to use the tool !!! note - Use the correct branch. - NebulaGraph 2.x and 3.x have different RPC protocols. + Use the correct branch. Different branches have different RPC protocols. 2. Access the directory `nebula-importer`. @@ -74,258 +81,281 @@ Configure the YAML file and prepare the CSV file to be imported to use the tool 4. Start the service. ```bash - $ ./nebula-importer --config + $ ./bin/nebula-importer --config ``` - !!! note - For details about the YAML configuration file, see configuration file description at the end of topic. - -### No network compilation mode - -If the server cannot be connected to the Internet, it is recommended to upload the source code and various dependency packages to the corresponding server for compilation on the machine that can be connected to the Internet. The operation steps are as follows: - -1. Clone repository. - - ```bash - $ git clone -b {{importer.branch}} https://github.com/vesoft-inc/nebula-importer.git - ``` - -2. Use the following command to download and package the dependent source code. - - ```bash - $ cd nebula-importer - $ go mod vendor - $ cd .. && tar -zcvf nebula-importer.tar.gz nebula-importer - ``` - -3. Upload the compressed package to a server that cannot be connected to the Internet. - -4. Unzip and compile. - - ```bash - $ tar -zxvf nebula-importer.tar.gz - $ cd nebula-importer - $ go build -mod vendor cmd/importer.go - ``` - ### Run in Docker mode Instead of installing the Go locale locally, you can use Docker to pull the [image](https://hub.docker.com/r/vesoft/nebula-importer) of the NebulaGraph Importer and mount the local configuration file and CSV data file into the container. The command is as follows: ```bash +$ docker pull vesoft/nebula-importer $ docker run --rm -ti \ - --network=host \ - -v : \ - -v : \ - vesoft/nebula-importer: - --config + --network=host \ + -v : \ + -v : \ + vesoft/nebula-importer: + --config ``` -- ``: The absolute path to the local YAML configuration file. -- ``: The absolute path to the local CSV data file. -- ``: NebulaGraph 2.x Please fill in 'v3'. +- ``: The absolute path to the YAML configuration file. +- ``: The absolute path to the CSV data file. If the file is not local, ignore this parameter. +- ``: NebulaGraph 3.x Please fill in 'v3'. !!! note A relative path is recommended. If you use a local absolute path, check that the path maps to the path in the Docker. ## Configuration File Description -NebulaGraph Importer uses configuration(`nebula-importer/examples/v2/example.yaml`) files to describe information about the files to be imported, the NebulaGraph server, and more. You can refer to the example configuration file: [Configuration without Header](config-without-header.md)/[Configuration with Header](config-with-header.md). This section describes the fields in the configuration file by category. +Various example configuration files are available within the [Github](https://github.com/vesoft-inc/nebula-ng-tools/tree/{{importer.branch}}/importer/examples) of the NebulaGraph Importer. The configuration files are used to describe information about the files to be imported, {{nebula.name}} server information, etc. The following section describes the fields within the configuration file in categories. !!! note If users download a binary package, create the configuration file manually. -### Basic configuration - -The example configuration is as follows: - -```yaml -version: v2 -description: example -removeTempFiles: false -``` - -|Parameter|Default value|Required|Description| -|:---|:---|:---|:---| -|`version`|v2|Yes|Target version of the configuration file.| -|`description`|example|No|Description of the configuration file.| -|`removeTempFiles`|false|No|Whether to delete temporarily generated logs and error data files.| - ### Client configuration -The client configuration stores the configurations associated with NebulaGraph. +Client configuration stores the configuration associated with the client's connection to the {{nebula.name}}. The example configuration is as follows: ```yaml -clientSettings: +client: + version: v3 + address: "192.168.1.100:9669,192.168.1.101:9669" + user: root + password: nebula + concurrencyPerAddress: 10 + reconnectInitialInterval: 1s retry: 3 - concurrency: 10 - channelBufferSize: 128 - space: test - connection: - user: user - password: password - address: 192.168.*.13:9669,192.168.*.14:9669 - postStart: - commands: | - UPDATE CONFIGS storage:wal_ttl=3600; - UPDATE CONFIGS storage:rocksdb_column_family_options = { disable_auto_compactions = true }; - afterPeriod: 8s - preStop: - commands: | - UPDATE CONFIGS storage:wal_ttl=86400; - UPDATE CONFIGS storage:rocksdb_column_family_options = { disable_auto_compactions = false }; + retryInitialInterval: 1s ``` |Parameter|Default value|Required|Description| |:---|:---|:---|:---| -|`clientSettings.retry`|3|No|Retry times of nGQL statement execution failures.| -|`clientSettings.concurrency`|10|No|Number of NebulaGraph client concurrency.| -|`clientSettings.channelBufferSize`|128|No|Cache queue size per NebulaGraph client.| -|`clientSettings.space`|-|Yes|Specifies the NebulaGraph space to import the data into. Do not import multiple spaces at the same time to avoid performance impact.| -|`clientSettings.connection.user`|-|Yes|NebulaGraph user name.| -|`clientSettings.connection.password`|-|Yes|The password for the NebulaGraph user name.| -|`clientSettings.connection.address`|-|Yes|Addresses and ports for all Graph services.| -|`clientSettings.postStart.commands`|-|No|Configure some of the operations to perform after connecting to the NebulaGraph server, and before inserting data.| -|`clientSettings.postStart.afterPeriod`|-|No|The interval, between executing the above `commands` and executing the insert data command, such as `8s`.| -|`clientSettings.preStop.commands`|-|No|Configure some of the actions you performed before disconnecting from the NebulaGraph server.| +|`client.version`|`v3`|Yes| Specifies the major version of the NebulaGraph. Currently only `v3` is supported.| +|`client.address`|`"127.0.0.1:9669"`|Yes| Specifies the address of the NebulaGraph. Multiple addresses are separated by commas.| +|`client.user`|`root`|No| NebulaGraph user name.| +|`client.password`|`nebula`|No| The password for the NebulaGraph user name.| +|`client.concurrencyPerAddress`|`10`|No| The number of concurrent client connections for a single graph service.| +|`client.retryInitialInterval`|`1s`|No| Reconnect interval time.| +|`client.retry`|`3`|No| The number of retries for failed execution of the nGQL statement.| +|`client.retryInitialInterval`|`1s`|No| Retry interval time.| -### File configuration +### Manager configuration -File configuration Stores the configuration of data files and logs, and details about the Schema. - -#### File and log configuration +Manager configuration is a human-controlled configuration after connecting to the database. The example configuration is as follows: ```yaml -workingDir: ./data/ -logPath: ./err/test.log -files: - - path: ./student.csv - failDataPath: ./err/student.csv - batchSize: 128 - limit: 10 - inOrder: false - type: csv - csv: - withHeader: false - withLabel: false - delimiter: "," - lazyQuotes: false +manager: + spaceName: basic_string_examples + batch: 128 + readerConcurrency: 50 + importerConcurrency: 512 + statsInterval: 10s + hooks: + before: + - statements: + - UPDATE CONFIGS storage:wal_ttl=3600; + - UPDATE CONFIGS storage:rocksdb_column_family_options = { disable_auto_compactions = true }; + - statements: + - | + DROP SPACE IF EXISTS basic_int_examples; + CREATE SPACE IF NOT EXISTS basic_int_examples(partition_num=5, replica_factor=1, vid_type=int); + USE basic_int_examples; + wait: 10s + after: + - statements: + - | + UPDATE CONFIGS storage:wal_ttl=86400; + UPDATE CONFIGS storage:rocksdb_column_family_options = { disable_auto_compactions = false }; ``` |Parameter|Default value|Required|Description| |:---|:---|:---|:---| -|`workingDir`|-|No|If you have multiple directories containing data with the same file structure, you can use this parameter to switch between them. For example, the value of `path` and `failDataPath` of the configuration below will be automatically changed to `./data/student.csv` and .`/data/err/student`. If you change `workingDir` to `./data1`, the path will be changed accordingly. The param can be either absolute or relative.| -|`logPath`|-|No|Path for exporting log information, such as errors during import.| -|`files.path`|-|Yes|Path for storing data files. If a relative path is used, the path is merged with the current configuration file directory. You can use an asterisk (\*) for fuzzy matching to import multiple files with similar names, but the files need to be the same structure.| -|`files.failDataPath`|-|Yes|Insert the failed data file storage path, so that data can be written later.| -|`files.batchSize`|128|No|The number of statements inserting data in a batch.| -|`files.limit`|-|No|Limit on the number of rows of read data.| -|`files.inOrder`|-|No|Whether to insert rows in the file in order. If the value is set to `false`, the import rate decreases due to data skew.| -|`files.type`|-|Yes|The file type.| -|`files.csv.withHeader`|`false`|Yes|Whether there is a header.| -|`files.csv.withLabel`|`false`|Yes|Whether there is a label.| -|`files.csv.delimiter`|`","`|Yes|Specifies the delimiter for the CSV file. A string delimiter that supports only one character.| -|`lazyQuotes`|`false`|No|If `lazyQuotes` is true, a quote may appear in an unquoted field and a non-doubled quote may appear in a quoted field.| - -#### Schema configuration +|`manager.spaceName`|-|Yes| Specifies the NebulaGraph space to import the data into. Do not support importing multiple map spaces at the same time.| +|`manager.batch`|`128`|No| The batch size for executing statements (global configuration). +
Setting the batch size individually for a data source can using the parameter `sources.batch` below.| +|`manager.readerConcurrency`|`50`|No| The number of concurrent reads of the data source by the reader.| +|`manager.importerConcurrency`|`512`|No| The number of concurrent nGQL statements generated to be executed, and then will call the client to execute these nGQL statements.| +|`manager.statsInterval`|`10s`|No| The time interval for printing statistical information| +|`manager.hooks.before.[].statements`|-|No|The command to execute in the graph space before importing.| +|`manager.hooks.before.[].wait`|-|No|The wait time after `statements` are executed.| +|`manager.hooks.after.[].statements`|-|No|The commands to execute in the graph space after importing.| +|`manager.hooks.after.[].wait`|-|No|The wait time after `statements` are executed.| -Schema configuration describes the Meta information of the current data file. Schema types are vertex and edge. Multiple vertexes or edges can be configured at the same time. +### Log configuration -- vertex configuration +Log configuration is the logging-related configuration. The example configuration is as follows: ```yaml -schema: - type: vertex - vertex: - vid: - index: 1 - function: hash - prefix: abc - tags: - - name: student - props: - - name: age - type: int - index: 2 - - name: name - type: string - index: 1 - - name: gender - type: string - - name: phone - type: string - nullable: true - - name: wechat - type: string - nullable: true - nullValue: "__NULL__" +log: + level: INFO + console: true + files: + - logs/nebula-importer.log ``` |Parameter|Default value|Required|Description| |:---|:---|:---|:---| -|`files.schema.type`|-|Yes|Schema type. Possible values are `vertex` and `edge`.| -|`files.schema.vertex.vid.index`|-|No|The vertex ID corresponds to the column number in the CSV file.| -|`files.schema.vertex.vid.function`|-|No|Functions to generate the VIDs. Currently, we only support function `hash`.| -|`files.schema.vertex.vid.prefix`|-|No|Add prefix to the original vid. When function is specified also, `prefix` is applied to the original vid before `function`.| -|`files.schema.vertex.tags.name`|-|Yes|Tag name.| -|`files.schema.vertex.tags.props.name`|-|Yes|Tag property name, which must match the Tag property in the NebulaGraph.| -|`files.schema.vertex.tags.props.type`|-|Yes|Property data type, supporting `bool`,`int`,`float`,`double`,`string`,`time`,`timestamp`,`date`,`datetime`,`geography`,`geography(point)`,`geography(linestring)` and `geography(polygon)`.| -|`files.schema.vertex.tags.props.index`|-|No|Property corresponds to the sequence number of the column in the CSV file.| -|`files.schema.vertex.tags.props.nullable`|`false`|No|Whether this prop property can be `NULL`, optional values is `true` or `false`.| -|`files.schema.vertex.tags.props.nullValue`|`""`|No|Ignored when nullable is false. The property is set to NULL when the value is equal to nullValue.| -|`files.schema.vertex.tags.props.alternativeIndices`|-|No| Ignored when `nullable` is `false`. When the property value is not `nullValue`, the value is fetched from csv according to the index sequence.| -|`files.schema.vertex.tags.props.defaultValue`|-|No| Ignored when `nullable` is false. The property default value, when all the values obtained by `index` and `alternativeIndices` are `nullValue`.| +|`log.level`|`INFO`|No| Specifies the log level. Optional values are `DEBUG`, `INFO`, `WARN`, `ERROR`, `PANIC`, `FATAL`.| +|`log.console`|`true`|No| Whether to print the logs to console synchronously when storing logs.| +|`log.files`|-|No|The log file path.| -!!! note - The sequence numbers of the columns in the CSV file start from 0, that is, the sequence numbers of the first column are 0, and the sequence numbers of the second column are 1. +### Source configuration -- edge configuration +The Source configuration requires configuration of data source information, data processing methods, and Schema mapping. The example configuration is as follows: ```yaml -schema: - type: edge - edge: - name: follow - srcVID: - index: 0 - function: hash - dstVID: - index: 1 - function: - rank: - index: 2 - props: - - name: grade - type: int - index: 3 +sources: + - path: ./person.csv # Required. Specifies the path where the data files are stored. If a relative path is used, the path and current configuration file directory are spliced. Wildcard filename is also supported, for example: ./follower-*.csv, please make sure that all matching files with the same schema. +# - s3: # AWS S3 +# endpoint: endpoint # Optional. The endpoint of S3 service, can be omitted if using AWS S3. +# region: us-east-1 # Required. The region of S3 service. +# bucket: gdelt-open-data # Required. The bucket of file in S3 service. +# key: events/20190918.export.csv # Required. The object key of file in S3 service. +# accessKeyID: "" # Optional. The access key of S3 service. If it is public data, no need to configure. +# accessKeySecret: "" # Optional. The secret key of S3 service. If it is public data, no need to configure. +# - oss: +# endpoint: https://oss-cn-hangzhou.aliyuncs.com # Required. The endpoint of OSS service. +# bucket: bucketName # Required. The bucket of file in OSS service. +# key: objectKey # Required. The object key of file in OSS service. +# accessKeyID: accessKey # Required. The access key of OSS service. +# accessKeySecret: secretKey # Required. The secret key of OSS service. +# - ftp: +# host: 192.168.0.10 # Required. The host of FTP service. +# port: 21 # Required. The port of FTP service. +# user: user # Required. The user of FTP service. +# password: password # Required. The password of FTP service. +# path: "/events/20190918.export.csv" # Required. The path of file in the FTP service. +# - sftp: +# host: 192.168.0.10 # Required. The host of SFTP service. +# port: 22 # Required. The port of SFTP service. +# user: user # Required. The user of SFTP service. +# password: password # Optional. The password of SFTP service. +# keyFile: keyFile # Optional. The ssh key file path of SFTP service. +# keyData: keyData $ Optional. The ssh key file content of SFTP service. +# passphrase: passphrase # Optional. The ssh key passphrase of SFTP service. +# path: "/events/20190918.export.csv" # Required. The path of file in the SFTP service. +# - hdfs: +# address: "127.0.0.1:8020" # Required. The address of HDFS service. +# user: "hdfs" # Optional. The user of HDFS service. +# path: "/events/20190918.export.csv" # Required. The path of file in the HDFS service. + batch: 256 + csv: + delimiter: "|" + withHeader: false + lazyQuotes: false + tags: + - name: Person + id: + type: "STRING" + function: "hash" +# index: 0 + concatItems: + - person_ + - 0 + - _id + props: + - name: "firstName" + type: "STRING" + index: 1 + - name: "lastName" + type: "STRING" + index: 2 + - name: "gender" + type: "STRING" + index: 3 + nullable: true + defaultValue: female + - name: "birthday" + type: "DATE" + index: 4 + nullable: true + nullValue: _NULL_ + - name: "creationDate" + type: "DATETIME" + index: 5 + - name: "locationIP" + type: "STRING" + index: 6 + - name: "browserUsed" + type: "STRING" + index: 7 + - path: ./knows.csv + batch: 256 + edges: + - name: KNOWS # person_knows_person + src: + id: + type: "STRING" + concatItems: + - person_ + - 0 + - _id + dst: + id: + type: "STRING" + concatItems: + - person_ + - 1 + - _id + props: + - name: "creationDate" + type: "DATETIME" + index: 2 + nullable: true + nullValue: _NULL_ + defaultValue: 0000-00-00T00:00:00 ``` -|Parameter|Default value|Required|Description| -|:---|:---|:---|:---| -|`files.schema.type`|-|Yes|Schema type. Possible values are `vertex` and `edge`.| -|`files.schema.edge.name`|-|Yes|Edge type name.| -|`files.schema.edge.srcVID.index`|-|No|The data type of the starting vertex ID of the edge.| -|`files.schema.edge.srcVID.function`|-|No|Functions to generate the source vertex. Currently, we only support function `hash`.| -|`files.schema.edge.dstVID.index`|-|No|The destination vertex ID of the edge corresponds to the column number in the CSV file.| -|`files.schema.edge.dstVID.function`|-|No|Functions to generate the destination vertex. Currently, we only support function `hash`.| -|`files.schema.edge.rank.index`|-|No|The rank value of the edge corresponds to the column number in the CSV file.| -|`files.schema.edge.props.name`|-|Yes|The Edge Type property name must match the Edge Type property in the NebulaGraph.| -|`files.schema.edge.props.type`|-|Yes|Property data type, supporting `bool`, `int`, `float`, `double`, `timestamp`, `string`, and `geo`.| -|`files.schema.edge.props.index`|-|No|Property corresponds to the sequence number of the column in the CSV file.| +The configuration mainly includes the following parts: -## About the CSV file header +- Specify the data source information. +- Specifies the batch size for executing statements. +- Specifies the CSV file format information. +- Specifies the schema mapping for Tag. +- Specifies the schema mapping for Edge type. -According to whether the CSV file has a header or not, the Importer needs to make different Settings on the configuration file. For relevant examples and explanations, please refer to: - -- [Configuration without Header](config-without-header.md) +|Parameter|Default value|Required|Description| +|:---|:---|:---|:---| +|`sources.path`
`sources.s3`
`sources.oss`
`sources.ftp`
`sources.sftp`
`sources.hdfs` |-| No | Specify data source information, such as local file, HDFS, and S3. Only one source can be configured for the `source`. Configure multiple sources in multiple `source`.
See the comments in the example for configuration items for different data sources. | +|`sources.batch` |`256`| No | The batch size for executing statements when importing this data source. The priority is higher than `manager.batch`. | +|`sources.csv.delimiter` |`,`| No | Specifies the delimiter for the CSV file. Only 1-character string separators are supported. When using special characters as separators, they need to be escaped. For example, when the delimiter is `0x03` in hexadecimal, i.e. `Ctrl+C`, the escape is written as `"\x03"` or `"\u0003"`. For details on escaping special characters in yaml format, see [Escaped Characters](https://yaml.org/spec/1.2.2/#escaped-characters).| | +|`sources.csv.withHeader` |`false`| No | Whether to ignore the first record in the CSV file. | +|`sources.csv.lazyQuotes` |`false`| No | Whether to allow lazy quotes. If `lazyQuotes` is true, a quote may appear in an unquoted field and a non-doubled quote may appear in a quoted field. | +|`sources.tags.name` |-| Yes | The tag name. | +|`sources.tags.id.type` |`STRING`| No | The type of the VID. | +|`sources.tags.id.function` |-| No | Functions to generate the VID. Currently, only function `hash` are supported. | +|`sources.tags.id.index` |-| No | The column number corresponding to the VID in the data file. If `sources.tags.id.concatItems` is not configured, this parameter must be configured. | +|`sources.tags.id.concatItems` |-| No | Used to concatenate two or more arrays, the concatenated items can be `string`, `int` or mixed. `string` stands for a constant, `int` for an index column. If this parameter is set, the `sources.tags.id.index` parameter will not take effect. | +|`sources.tags.ignoreExistedIndex` |`true`| No | Whether to enable `IGNORE_EXISTED_INDEX`, that is, do not update index after insertion vertex. | +|`sources.tags.props.name` |-| Yes | The tag property name, which must match the Tag property in the database. | +|`sources.tags.props.type` |`STRING`| No | Property data type, supporting `BOOL`, `INT`, `FLOAT`, `DOUBLE`, `STRING`, `TIME`, `TIMESTAMP`, `DATE`, `DATETIME`, `GEOGRAPHY`, `GEOGRAPHY(POINT)`, `GEOGRAPHY(LINESTRING)` and `GEOGRAPHY(POLYGON)`. | +|`sources.tags.props.index` |-| Yes | The property corresponds to the column number in the data file. | +|`sources.tags.props.nullable` |`false`| No | Whether this prop property can be `NULL`, optional values is `true` or `false`. | +|`sources.tags.props.nullValue` |-| No | Ignored when `nullable` is `false`. The value used to determine whether it is a `NULL`. The property is set to `NULL` when the value is equal to `nullValue`. | +|`sources.tags.props.alternativeIndices` |-| No | Ignored when `nullable` is `false`. The property is fetched from records according to the indices in order until not equal to `nullValue`. | +|`sources.tags.props.defaultValue` |-| No | Ignored when `nullable` is `false`. The property default value, when all the values obtained by `index` and `alternativeIndices` are `nullValue`. | +|`sources.edges.name` |-| Yes | The edge type name. | +|`sources.edges.src.id.type` |`STRING`| No | The data type of the VID at the starting vertex on the edge. | +|`sources.edges.src.id.index` |-| Yes | The column number in the data file corresponding to the VID at the starting vertex on the edge. | +|`sources.edges.dst.id.type` |`STRING`| No | The data type of the VID at the destination vertex on the edge. | +|`sources.edges.dst.id.index` |-| Yes | The column number in the data file corresponding to the VID at the destination vertex on the edge. | +|`sources.edges.rank.index` |-| No | The column number in the data file corresponding to the rank on the edge. | +|`sources.edges.ignoreExistedIndex` |`true`| No | Whether to enable `IGNORE_EXISTED_INDEX`, that is, do not update index after insertion vertex. | +|`sources.edges.props.name` |-| Yes | The edge type property name, which must match the Tag property in the database. | +|`sources.edges.props.type` |`STRING`| No | Property data type, supporting `BOOL`, `INT`, `FLOAT`, `DOUBLE`, `STRING`, `TIME`, `TIMESTAMP`, `DATE`, `DATETIME`, `GEOGRAPHY`, `GEOGRAPHY(POINT)`, `GEOGRAPHY(LINESTRING)` and `GEOGRAPHY(POLYGON)`. | +|`sources.edges.props.index` |-| Yes | The property corresponds to the column number in the data file. | +|`sources.edges.props.nullable` |-| No | Whether this prop property can be `NULL`, optional values is `true` or `false`. | +|`sources.edges.props.nullValue` |-| No | Ignored when `nullable` is `false`. The value used to determine whether it is a `NULL`. The property is set to `NULL` when the value is equal to `nullValue`. | +|`sources.edges.props.defaultValue` |-| No | Ignored when `nullable` is `false`. The property default value, when all the values obtained by `index` and `alternativeIndices` are `nullValue`. | -- [Configuration with Header](config-with-header.md) +!!! note + The sequence numbers of the columns in the CSV file start from 0, that is, the sequence numbers of the first column are 0, and the sequence numbers of the second column are 1. diff --git a/mkdocs.yml b/mkdocs.yml index 1fe9091709b..26ec661f9ef 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -79,6 +79,8 @@ plugins: - 4.deployment-and-installation/3.upgrade-nebula-graph/upgrade-nebula-from-200-to-latest.md - nebula-cloud/* - nebula-cloud.md + - nebula-importer/config-with-header.md + - nebula-importer/config-without-header.md # When publishing a version of a document that includes Enterprise Edition, annotation the following page # ent.begin # - 3.ngql-guide/6.functions-and-expressions/17.ES-function.md @@ -674,8 +676,8 @@ nav: - NebulaGraph Importer: - Use NebulaGraph Importer: nebula-importer/use-importer.md - - Configuration with Header: nebula-importer/config-with-header.md - - Configuration without Header: nebula-importer/config-without-header.md +# - Configuration with Header: nebula-importer/config-with-header.md +# - Configuration without Header: nebula-importer/config-without-header.md - NebulaGraph Exchange: - Introduction: