From 76da913b2c8b7f56f526a01e62a0c0d5c9d0422c Mon Sep 17 00:00:00 2001 From: foesa <35463247+foesa-yang@users.noreply.github.com> Date: Tue, 7 Feb 2023 15:23:55 +0800 Subject: [PATCH 1/2] Update full text index --- .../14.native-index-statements/README.md | 10 -- .../1.search-with-text-based-index.md | 15 +-- .../17.ES-function.md | 106 ------------------ .../1.text-based-index-restrictions.md | 20 +--- .../6.deploy-text-based-index/2.deploy-es.md | 62 ++-------- .../3.deploy-listener.md | 8 -- mkdocs.yml | 2 - 7 files changed, 15 insertions(+), 208 deletions(-) delete mode 100644 docs-2.0/3.ngql-guide/6.functions-and-expressions/17.ES-function.md diff --git a/docs-2.0/3.ngql-guide/14.native-index-statements/README.md b/docs-2.0/3.ngql-guide/14.native-index-statements/README.md index 293d16ab75d..fa9673f2078 100644 --- a/docs-2.0/3.ngql-guide/14.native-index-statements/README.md +++ b/docs-2.0/3.ngql-guide/14.native-index-statements/README.md @@ -38,22 +38,12 @@ Full-text indexes are used to do prefix, wildcard, regexp, and fuzzy search on a - Full-text indexes allow indexing just one property. -- Only strings within a specified length (no longer than 256 bytes) are indexed. - - Full-text indexes do not support logical operations such as `AND`, `OR`, and `NOT`. !!! Note To do complete string matches, use native indexes. -### Operations on full-text indexes - -Before doing any operations on full-text indexes, please make sure that you deploy full-text indexes. Details on full-text indexes deployment, see [Deploy Elasticsearch](../../4.deployment-and-installation/6.deploy-text-based-index/2.deploy-es.md) and [Deploy Listener](../../4.deployment-and-installation/6.deploy-text-based-index/3.deploy-listener.md). - -At this time, full-text indexes are created automatically on the Elasticsearch cluster. And rebuilding or altering full-text indexes are not supported. To drop full-text indexes, you need to drop them on the Elasticsearch cluster manually. - -To query full-text indexes, see [Search with full-text indexes](../15.full-text-index-statements/1.search-with-text-based-index.md). - ## Null values Indexes do not support indexing null values. diff --git a/docs-2.0/3.ngql-guide/15.full-text-index-statements/1.search-with-text-based-index.md b/docs-2.0/3.ngql-guide/15.full-text-index-statements/1.search-with-text-based-index.md index b1ed380e00b..b2a456a1172 100644 --- a/docs-2.0/3.ngql-guide/15.full-text-index-statements/1.search-with-text-based-index.md +++ b/docs-2.0/3.ngql-guide/15.full-text-index-statements/1.search-with-text-based-index.md @@ -14,14 +14,14 @@ Before using the full-text index, make sure that you know the [restrictions](../ ## Natural language full-text search -A natural language search interprets the search string as a phrase in natural human language. The search is case-insensitive. By default, each substring (separated by spaces) will be searched separately. For example, there are three vertices with the tag `player`. The tag `player` contains the property `name`. The `name` of these three vertices are `Kevin Durant`, `Tim Duncan`, and `David Beckham`. Now that the full-text index of `player.name` is established, these three vertices will be queried when using the prefix search statement `LOOKUP ON player WHERE PREFIX(player.name,"d");`. +A natural language search interprets the search string as a phrase in natural human language. The search is case-sensitive and by default prefixes the string with a match. For example, there are three vertices with the tag `player`. The tag `player` contains the property `name`. The `name` of these three vertices are `Kevin Durant`, `Tim Duncan`, and `David Beckham`. Now that the full-text index of `player.name` is established, only `David Beckham` will be queried when using the prefix search statement `LOOKUP ON player WHERE PREFIX(player.name,"D");`. ## Syntax ### Create full-text indexes ```ngql -CREATE FULLTEXT {TAG | EDGE} INDEX ON { | } ([]); +CREATE FULLTEXT {TAG | EDGE} INDEX ON { | } ([]); ``` ### Show full-text indexes @@ -36,6 +36,9 @@ SHOW FULLTEXT INDEXES; REBUILD FULLTEXT INDEX; ``` +!!! caution + When there is a large amount of data, rebuilding full-text index is slow, you can modify `snapshot_send_files=false` in the configuration file of Storage service(`nebula-storaged.conf`). + ### Drop full-text indexes ```ngql @@ -94,13 +97,7 @@ nebula> SHOW LISTENER; // This example creates the tag. nebula> CREATE TAG IF NOT EXISTS player(name string, age int); -// This example creates the native index. -nebula> CREATE TAG INDEX IF NOT EXISTS name ON player(name(20)); - -// This example rebuilds the native index. -nebula> REBUILD TAG INDEX; - -// This example creates the full-text index. The index name starts with "nebula". +// This example creates the full-text index. The index name starts with "nebula_". nebula> CREATE FULLTEXT TAG INDEX nebula_index_1 ON player(name); // This example rebuilds the full-text index. diff --git a/docs-2.0/3.ngql-guide/6.functions-and-expressions/17.ES-function.md b/docs-2.0/3.ngql-guide/6.functions-and-expressions/17.ES-function.md deleted file mode 100644 index e0006625c7b..00000000000 --- a/docs-2.0/3.ngql-guide/6.functions-and-expressions/17.ES-function.md +++ /dev/null @@ -1,106 +0,0 @@ -# Elasticsearch query functions - -The Elasticsearch query functions allow NebulaGraph to send a GET request to Elasticsearch to read the data and convert the returned JSON into a Map format in NebulaGraph. Users can use the map data in the same composite graph query statement. - -For more Information about Elasticsearch syntax, see [Elasticsearch official document](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-your-data.html). - -!!! enterpriseonly - - Only available for the NebulaGraph Enterprise Edition. - -## Precautions - -- You can not use the Elasticsearch service that a full-text index has been deployed. You need to use the independent Elasticsearch service and write data to the Elasticsearch service by yourself. NebulaGraph will not write data to the Elasticsearch. To deploy an Elasticsearch cluster, see [Kubernetes Elasticsearch deployment](https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-deploy-elasticsearch.html) or [Elasticsearch installation](https://www.elastic.co/guide/en/elasticsearch/reference/7.15/targz.html). - -- `SIGN IN` is required before use the Elasticsearch query functions, for example, `SIGN IN TEXT SERVICE (127.0.0.1:9200, HTTP)`. You can `SIGN OUT` the service when it is no longer needed. For more Information, see [Sign in to the text search clients](../../4.deployment-and-installation/6.deploy-text-based-index/2.deploy-es.md#_sign_in_to_the_text_search_clients). - -- The Elasticsearch query functions support only GET requests. Perform the POST, PUT, and DELETE requests should be called by the Elasticsearch client. - -- The Elasticsearch query functions have no timeout, NebulaGraph will wait for Elasticsearch to return results. - -- Using operator directly after the Elasticsearch query functions is not supported. For example, `RETURN calles("/test/player/1")["_source"]` is not allowed. - -- Compatible with Elasticsearch 7 and later, not tested for earlier versions. - -## OpenCypher-compatible - -This topic applies to the openCypher syntax in nGQL only. - -## Syntax - -Users can use calles() to execute am Elasticsearch query, and also use callesfile() to execute am Elasticsearch query in a specified file. - -```ngql -calles(""); -callesfile(""); -``` - -- Result type: Map. Elasticsearch returns a string in JSON format, which NebulaGraph converts to map format. - -## Example - -```ngql -nebula> RETURN calles("/test/player/1") AS a; -+---------------------------------------------------------------------------------------------------------------------------------------------+ -| a | -+---------------------------------------------------------------------------------------------------------------------------------------------+ -| {_id: "1", _index: "test", _primary_term: 1, _seq_no: 0, _source: {age: 43, name: "Tim Duncan"}, _type: "player", _version: 1, found: true} | -+---------------------------------------------------------------------------------------------------------------------------------------------+ - -nebula> WITH calles("/test/player/1") AS a RETURN a; -+---------------------------------------------------------------------------------------------------------------------------------------------+ -| a | -+---------------------------------------------------------------------------------------------------------------------------------------------+ -| {_id: "1", _index: "test", _primary_term: 1, _seq_no: 0, _source: {age: 43, name: "Tim Duncan"}, _type: "player", _version: 1, found: true} | -+---------------------------------------------------------------------------------------------------------------------------------------------+ - -nebula> WITH calles("/test/player/1") AS a \ - MATCH (b:player) WHERE b.player.age == a["_source"]["age"] \ - RETURN b; -+---------------------------------------------------+ -| b | -+---------------------------------------------------+ -| ("player141" :player{age: 43, name: "Ray Allen"}) | -+---------------------------------------------------+ - -//For non-existent data, the found field returns false. -nebula> WITH calles("/test/player/123") AS a RETURN a; -+-------------------------------------------------------------+ -| a | -+-------------------------------------------------------------+ -| {_id: "123", _index: "test", _type: "player", found: false} | -+-------------------------------------------------------------+ - -//For non-existent file, NebulaGraph returns an error。 -nebula> RETURN callesfile('/home/xxx/es_query_test.txt'); -+---------------------------------------------+ -| callESFile( "/home/xxx/es_query_test.txt" ) | -+---------------------------------------------+ -| {error: "read Elaticsearch file fail."} | -+---------------------------------------------+ - -nebula> RETURN callesfile('/home/xxx/es_query.txt'); -+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ -| callESFile( "/home/xxx/es_query.txt" ) | -+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ -| {_shards: {failed: 0, skipped: 0, successful: 1, total: 1}, hits: {hits: [{_id: "2", _index: "test", _score: 1.0, _source: {age: 36, name: "Tony Parker"}... | -+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ - - -nebula> WITH callesfile('/home/xxx/es_query.txt') AS s RETURN [n in s["hits"]["hits"]|n.`_source`.name]; -+--------------------------------------------------------------------------------------------------------------------------------------+ -| [n IN s["hits"]["hits"] | n._source.name] | -+--------------------------------------------------------------------------------------------------------------------------------------+ -| ["LeBron James", "Danny Green", "Kevin Durant", "Stephen Curry", "Rudy Gay", "Ray Allen", "Tim Duncan", "Tony Parker", "Chris Paul"] | -+--------------------------------------------------------------------------------------------------------------------------------------+ - -nebula> WITH callesfile('/home/xxx/es_query.txt') AS s \ - MATCH p= (a:player)-[:serve]->() \ - WHERE id(a) in [n in s["hits"]["hits"]|n.`_source`.name] RETURN p; -+---------------------------------------------------------------------------------------------------------------------------------------------------+ -| p | -+---------------------------------------------------------------------------------------------------------------------------------------------------+ -| <("LeBron James" :player{age: 34, name: "LeBron James"})-[:serve@1 {end_year: 2018, start_year: 2014}]->("Cavaliers" :team{name: "Cavaliers"})> | -| <("LeBron James" :player{age: 34, name: "LeBron James"})-[:serve@0 {end_year: 2010, start_year: 2003}]->("Cavaliers" :team{name: "Cavaliers"})> | -... -``` diff --git a/docs-2.0/4.deployment-and-installation/6.deploy-text-based-index/1.text-based-index-restrictions.md b/docs-2.0/4.deployment-and-installation/6.deploy-text-based-index/1.text-based-index-restrictions.md index b578650e65a..26bd6df5f5f 100644 --- a/docs-2.0/4.deployment-and-installation/6.deploy-text-based-index/1.text-based-index-restrictions.md +++ b/docs-2.0/4.deployment-and-installation/6.deploy-text-based-index/1.text-based-index-restrictions.md @@ -8,17 +8,11 @@ For now, full-text search has the following limitations: - Currently, full-text search supports `LOOKUP` statements only. -- Uppercase letters are not allowed in a full-text index name. - -- The maximum indexing string length is 256 bytes. The part of data that exceeds 256 bytes will not be indexed. - -- Full-text search is native-index-dependent. You need to first create an index for tags/edge types and then you can use the full-text index feature. +- The full-text index name must starts with `nebula_` and can contain only numbers, lowercase letters, underscores, and strings no longer than 256 bytes. - If there is a full-text index on the tag/edge type, the tag/edge type cannot be deleted or modified. -- One tag/edge type can only have one full-text index. - -- The type of properties must be `string`. +- The type of properties must be `STRING` or `FIXED_STRING`. - Full-text index can not be applied to search multiple tags/edge types. @@ -26,24 +20,14 @@ For now, full-text search has the following limitations: - Full-text index can not search properties with value `NULL`. -- Elastic Search creates the corresponding index automatically when writing data. The error `text search not found` will be displayed when users created a full-text index but no corresponding data was written. - - Altering Elasticsearch indexes is not supported at this time. - The pipe operator is not supported. - `WHERE` clauses supports full-text search only working on single terms. -- Full-text indexes are not deleted together with the graph space. - - Make sure that you start the Elasticsearch cluster and Nebula Graph at the same time. If not, the data writing on the Elasticsearch cluster can be incomplete. -- Do not contain `'` or `\` in the vertex or edge values. If not, an error will be caused in the Elasticsearch cluster storage. - - It may take a while for Elasticsearch to create indexes. If Nebula Graph warns no index is found, wait for the index to take effect (however, the waiting time is unknown and there is no code to check). - NebulaGraph clusters deployed with K8s do not support the full-text search feature. - -{{ ent.ent_begin }} -- Can not be used together with [calles()](../../3.ngql-guide/6.functions-and-expressions/17.ES-function.md) function. -{{ ent.ent_end }} \ No newline at end of file diff --git a/docs-2.0/4.deployment-and-installation/6.deploy-text-based-index/2.deploy-es.md b/docs-2.0/4.deployment-and-installation/6.deploy-text-based-index/2.deploy-es.md index 7e9cd890528..740d643f50c 100644 --- a/docs-2.0/4.deployment-and-installation/6.deploy-text-based-index/2.deploy-es.md +++ b/docs-2.0/4.deployment-and-installation/6.deploy-text-based-index/2.deploy-es.md @@ -10,65 +10,13 @@ Before you start using the full-text index, please make sure that you know the [ To deploy an Elasticsearch cluster, see [Kubernetes Elasticsearch deployment](https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-deploy-elasticsearch.html) or [Elasticsearch installation](https://www.elastic.co/guide/en/elasticsearch/reference/7.15/targz.html). -When the Elasticsearch cluster is started, add the template file for the Nebula Graph full-text index. - -!!! note - - The following template works for ES 7.8. The template content may vary depending on different ES versions. Search [Elasticsearch documents](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-templates.html) for the template corresponding to your ES version. - -```json -{ - "template": "nebula*", - "settings": { - "index": { - "number_of_shards": 3, - "number_of_replicas": 1 - } - }, - "mappings": { - "properties" : { - "tag_id" : { "type" : "long" }, - "column_id" : { "type" : "text" }, - "value" :{ "type" : "keyword"} - } - } -} -``` - -Make sure that you specify the following fields in strict accordance with the preceding template format: +!!! compatibility -```json -"template": "nebula*" -"tag_id" : { "type" : "long" }, -"column_id" : { "type" : "text" }, -"value" :{ "type" : "keyword"} -``` + NebulaGraph 3.4 and later, no additional templates need to be created. !!! caution - When creating a full-text index, start the index name with `nebula`. - -For example: - -```bash -curl -H "Content-Type: application/json; charset=utf-8" -XPUT http://127.0.0.1:9200/_template/nebula_index_template -d ' -{ - "template": "nebula*", - "settings": { - "index": { - "number_of_shards": 3, - "number_of_replicas": 1 - } - }, - "mappings": { - "properties" : { - "tag_id" : { "type" : "long" }, - "column_id" : { "type" : "text" }, - "value" :{ "type" : "keyword"} - } - } -}' -``` + The full-text index name must starts with `nebula_`. You can configure the Elasticsearch to meet your business needs. To customize the Elasticsearch, see [Elasticsearch Document](https://www.elastic.co/guide/en/elasticsearch/reference/current/settings.html). @@ -92,6 +40,10 @@ nebula> SIGN IN TEXT SERVICE (127.0.0.1:9200, HTTP); Elasticsearch does not have a username or password by default. If you configured a username and password, you need to specify them in the `SIGN IN` statement. +!!! caution + + The Elasticsearch client can only be logged in once, and if there are changes, you need to `SIGN OUT` and then `SIGN IN` again, and the client takes effect globally, and multiple graph spaces share the same Elasticsearch client. + ## Show text search clients The `SHOW TEXT SEARCH CLIENTS` statement can list the text search clients. diff --git a/docs-2.0/4.deployment-and-installation/6.deploy-text-based-index/3.deploy-listener.md b/docs-2.0/4.deployment-and-installation/6.deploy-text-based-index/3.deploy-listener.md index 20d567f9cdf..f150e6c2fbc 100644 --- a/docs-2.0/4.deployment-and-installation/6.deploy-text-based-index/3.deploy-listener.md +++ b/docs-2.0/4.deployment-and-installation/6.deploy-text-based-index/3.deploy-listener.md @@ -104,11 +104,3 @@ Run the `REMOVE LISTENER ELASTICSEARCH` statement to remove all Listeners in a g ```ngql nebula> REMOVE LISTENER ELASTICSEARCH; ``` - -!!! danger - - After the Listener is deleted, it cannot be added again. Therefore, the synchronization to the ES cluster cannot be continued and the text index data will be incomplete. If needed, you can only recreate the graph space. - -## Next - -After deploying the [Elasticsearch cluster](2.deploy-es.md) and the Listener, full-text indexes are created automatically on the Elasticsearch cluster. Users can do full-text search now. For more information, see [Full-Text search](../../3.ngql-guide/15.full-text-index-statements/1.search-with-text-based-index.md). diff --git a/mkdocs.yml b/mkdocs.yml index 27bc0c5134a..5c002c74ac5 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -332,8 +332,6 @@ nav: - Conditional expressions: 3.ngql-guide/6.functions-and-expressions/5.conditional-expressions.md - Predicate functions: 3.ngql-guide/6.functions-and-expressions/8.predicate.md - Geography functions: 3.ngql-guide/6.functions-and-expressions/14.geo.md -#ent - - Elasticsearch functions: 3.ngql-guide/6.functions-and-expressions/17.ES-function.md - User-defined functions: 3.ngql-guide/6.functions-and-expressions/9.user-defined-functions.md - General queries statements: From d63831443e699de4e7fdb538e7df7464ec4f4d67 Mon Sep 17 00:00:00 2001 From: foesa <35463247+foesa-yang@users.noreply.github.com> Date: Tue, 7 Feb 2023 15:45:11 +0800 Subject: [PATCH 2/2] Update docs-2.0/4.deployment-and-installation/6.deploy-text-based-index/2.deploy-es.md Co-authored-by: abby.huang <78209557+abby-cyber@users.noreply.github.com> --- .../6.deploy-text-based-index/2.deploy-es.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs-2.0/4.deployment-and-installation/6.deploy-text-based-index/2.deploy-es.md b/docs-2.0/4.deployment-and-installation/6.deploy-text-based-index/2.deploy-es.md index 740d643f50c..6a84b25a6a4 100644 --- a/docs-2.0/4.deployment-and-installation/6.deploy-text-based-index/2.deploy-es.md +++ b/docs-2.0/4.deployment-and-installation/6.deploy-text-based-index/2.deploy-es.md @@ -12,7 +12,7 @@ To deploy an Elasticsearch cluster, see [Kubernetes Elasticsearch deployment](ht !!! compatibility - NebulaGraph 3.4 and later, no additional templates need to be created. + For NebulaGraph 3.4 and later versions, no additional templates need to be created. !!! caution