From 4fcaabb37c7fc531af0fb023aab992925f6a2408 Mon Sep 17 00:00:00 2001 From: Stephen Crawford Date: Tue, 6 Jun 2023 12:52:31 -0400 Subject: [PATCH 01/16] Update network schema Signed-off-by: Stephen Crawford --- CHANGELOG.md | 3 ++- buildSrc/build.gradle | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 32cafe50f1b61..428a8782489a0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -46,6 +46,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), - Add `com.github.luben:zstd-jni` version 1.5.5-3 ([#2996](https://github.com/opensearch-project/OpenSearch/pull/2996)) - OpenJDK Update (April 2023 Patch releases) ([#7344](https://github.com/opensearch-project/OpenSearch/pull/7344) - Bump `com.amazonaws` 1.12.270 to `software.amazon.awssdk` 2.20.55 ([7372](https://github.com/opensearch-project/OpenSearch/pull/7372/)) +- Bump `com.networknt:json-schema-validator` from 1.0.81 to 1.0.83 ### Changed - [CCR] Add getHistoryOperationsFromTranslog method to fetch the history snapshot from translogs ([#3948](https://github.com/opensearch-project/OpenSearch/pull/3948)) @@ -102,7 +103,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), ### Removed ### Fixed -- Fixing error: adding a new/forgotten parameter to the configuration for checking the config on startup in plugins/repository-s3 #7924 +- Fixing error: adding a new/forgotten parameter to the configuration for checking the config on startup in plugins/repository-s3 #7924 ### Security diff --git a/buildSrc/build.gradle b/buildSrc/build.gradle index b58652e599ec8..8f3fcc64d89ca 100644 --- a/buildSrc/build.gradle +++ b/buildSrc/build.gradle @@ -118,7 +118,7 @@ dependencies { api 'com.avast.gradle:gradle-docker-compose-plugin:0.16.12' api "org.yaml:snakeyaml:${props.getProperty('snakeyaml')}" api 'org.apache.maven:maven-model:3.9.2' - api 'com.networknt:json-schema-validator:1.0.81' + api 'com.networknt:json-schema-validator:1.0.83' api "com.fasterxml.jackson.core:jackson-databind:${props.getProperty('jackson_databind')}" testFixturesApi "junit:junit:${props.getProperty('junit')}" From 9f9d7b338781fd653869cb1a6601380ba0dfec05 Mon Sep 17 00:00:00 2001 From: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> Date: Tue, 6 Jun 2023 12:55:27 -0400 Subject: [PATCH 02/16] Update CHANGELOG.md Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 428a8782489a0..ed2cd91f1e34f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -46,7 +46,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), - Add `com.github.luben:zstd-jni` version 1.5.5-3 ([#2996](https://github.com/opensearch-project/OpenSearch/pull/2996)) - OpenJDK Update (April 2023 Patch releases) ([#7344](https://github.com/opensearch-project/OpenSearch/pull/7344) - Bump `com.amazonaws` 1.12.270 to `software.amazon.awssdk` 2.20.55 ([7372](https://github.com/opensearch-project/OpenSearch/pull/7372/)) -- Bump `com.networknt:json-schema-validator` from 1.0.81 to 1.0.83 +- Bump `com.networknt:json-schema-validator` from 1.0.81 to 1.0.83 ([7833](https://github.com/opensearch-project/OpenSearch/pull/7933/)) ### Changed - [CCR] Add getHistoryOperationsFromTranslog method to fetch the history snapshot from translogs ([#3948](https://github.com/opensearch-project/OpenSearch/pull/3948)) From 111c815cb8b3065d1c012301ce939fcd8a5bf768 Mon Sep 17 00:00:00 2001 From: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> Date: Tue, 6 Jun 2023 12:55:48 -0400 Subject: [PATCH 03/16] Update CHANGELOG.md Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ed2cd91f1e34f..4aff05e939ac4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -46,7 +46,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), - Add `com.github.luben:zstd-jni` version 1.5.5-3 ([#2996](https://github.com/opensearch-project/OpenSearch/pull/2996)) - OpenJDK Update (April 2023 Patch releases) ([#7344](https://github.com/opensearch-project/OpenSearch/pull/7344) - Bump `com.amazonaws` 1.12.270 to `software.amazon.awssdk` 2.20.55 ([7372](https://github.com/opensearch-project/OpenSearch/pull/7372/)) -- Bump `com.networknt:json-schema-validator` from 1.0.81 to 1.0.83 ([7833](https://github.com/opensearch-project/OpenSearch/pull/7933/)) +- Bump `com.networknt:json-schema-validator` from 1.0.81 to 1.0.83 ([7933](https://github.com/opensearch-project/OpenSearch/pull/7933/)) ### Changed - [CCR] Add getHistoryOperationsFromTranslog method to fetch the history snapshot from translogs ([#3948](https://github.com/opensearch-project/OpenSearch/pull/3948)) From 966b7d97c871cafad6adeac2bdad697919613cdd Mon Sep 17 00:00:00 2001 From: Stephen Crawford Date: Wed, 7 Jun 2023 14:54:45 -0400 Subject: [PATCH 04/16] Add joni and jcoding to path Signed-off-by: Stephen Crawford --- buildSrc/build.gradle | 2 ++ .../gradle/precommit/ValidateJsonAgainstSchemaTask.java | 5 +++-- .../src/main/resources/rest-api-spec/api/bulk.json | 6 +----- 3 files changed, 6 insertions(+), 7 deletions(-) diff --git a/buildSrc/build.gradle b/buildSrc/build.gradle index 8f3fcc64d89ca..bacba8b367ddb 100644 --- a/buildSrc/build.gradle +++ b/buildSrc/build.gradle @@ -119,6 +119,8 @@ dependencies { api "org.yaml:snakeyaml:${props.getProperty('snakeyaml')}" api 'org.apache.maven:maven-model:3.9.2' api 'com.networknt:json-schema-validator:1.0.83' + api 'org.jruby.jcodings:jcodings:1.0.58' + api 'org.jruby.joni:joni:2.1.48' api "com.fasterxml.jackson.core:jackson-databind:${props.getProperty('jackson_databind')}" testFixturesApi "junit:junit:${props.getProperty('junit')}" diff --git a/buildSrc/src/main/java/org/opensearch/gradle/precommit/ValidateJsonAgainstSchemaTask.java b/buildSrc/src/main/java/org/opensearch/gradle/precommit/ValidateJsonAgainstSchemaTask.java index ff28197feb02b..fdf9e074db804 100644 --- a/buildSrc/src/main/java/org/opensearch/gradle/precommit/ValidateJsonAgainstSchemaTask.java +++ b/buildSrc/src/main/java/org/opensearch/gradle/precommit/ValidateJsonAgainstSchemaTask.java @@ -103,6 +103,7 @@ public void validate(InputChanges inputChanges) throws IOException { File jsonSchemaOnDisk = getJsonSchema(); getLogger().debug("JSON schema : [{}]", jsonSchemaOnDisk.getAbsolutePath()); SchemaValidatorsConfig config = new SchemaValidatorsConfig(); + config.setEcma262Validator(true); JsonSchemaFactory factory = JsonSchemaFactory.getInstance(SpecVersion.VersionFlag.V7); JsonSchema jsonSchema = factory.getSchema(mapper.readTree(jsonSchemaOnDisk), config); Map> errors = new LinkedHashMap<>(); @@ -113,7 +114,7 @@ public void validate(InputChanges inputChanges) throws IOException { File file = fileChange.getFile(); if (file.isDirectory() == false) { // validate all files and hold on to errors for a complete report if there are failures - getLogger().debug("Validating JSON [{}]", file.getName()); + getLogger().debug("Validating JSON: " + file.getName()); try { Set validationMessages = jsonSchema.validate(mapper.readTree(file)); maybeLogAndCollectError(validationMessages, errors, file); @@ -143,7 +144,7 @@ public void validate(InputChanges inputChanges) throws IOException { } } - private void maybeLogAndCollectError(Set messages, Map> errors, File file) { + private void maybeLogAndCollectError(Set messages, Map> errors, File file) throws IOException { for (ValidationMessage message : messages) { getLogger().error("[validate JSON][ERROR][{}][{}]", file.getName(), message.toString()); errors.computeIfAbsent(file, k -> new LinkedHashSet<>()) diff --git a/rest-api-spec/src/main/resources/rest-api-spec/api/bulk.json b/rest-api-spec/src/main/resources/rest-api-spec/api/bulk.json index bb066cd131480..978d3740524a9 100644 --- a/rest-api-spec/src/main/resources/rest-api-spec/api/bulk.json +++ b/rest-api-spec/src/main/resources/rest-api-spec/api/bulk.json @@ -36,11 +36,7 @@ }, "refresh":{ "type":"enum", - "options":[ - "true", - "false", - "wait_for" - ], + "options": ["true", "false", "wait_for"], "description":"If `true` then refresh the affected shards to make this operation visible to search, if `wait_for` then wait for a refresh to make this operation visible to search, if `false` (the default) then do nothing with refreshes." }, "routing":{ From 16e87767eb7b5b6d794ea33820846c2dbe09bfd6 Mon Sep 17 00:00:00 2001 From: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> Date: Wed, 7 Jun 2023 15:01:14 -0400 Subject: [PATCH 05/16] Update buildSrc/src/main/java/org/opensearch/gradle/precommit/ValidateJsonAgainstSchemaTask.java Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> --- .../gradle/precommit/ValidateJsonAgainstSchemaTask.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/buildSrc/src/main/java/org/opensearch/gradle/precommit/ValidateJsonAgainstSchemaTask.java b/buildSrc/src/main/java/org/opensearch/gradle/precommit/ValidateJsonAgainstSchemaTask.java index fdf9e074db804..72019f87d1ae6 100644 --- a/buildSrc/src/main/java/org/opensearch/gradle/precommit/ValidateJsonAgainstSchemaTask.java +++ b/buildSrc/src/main/java/org/opensearch/gradle/precommit/ValidateJsonAgainstSchemaTask.java @@ -114,7 +114,7 @@ public void validate(InputChanges inputChanges) throws IOException { File file = fileChange.getFile(); if (file.isDirectory() == false) { // validate all files and hold on to errors for a complete report if there are failures - getLogger().debug("Validating JSON: " + file.getName()); + getLogger().debug("Validating JSON [{}]", file.getName()); try { Set validationMessages = jsonSchema.validate(mapper.readTree(file)); maybeLogAndCollectError(validationMessages, errors, file); From f3bf8c59d7bc8a940889e1aaca8ea9cfab39bb78 Mon Sep 17 00:00:00 2001 From: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> Date: Wed, 7 Jun 2023 15:01:46 -0400 Subject: [PATCH 06/16] Update buildSrc/src/main/java/org/opensearch/gradle/precommit/ValidateJsonAgainstSchemaTask.java Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> --- .../gradle/precommit/ValidateJsonAgainstSchemaTask.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/buildSrc/src/main/java/org/opensearch/gradle/precommit/ValidateJsonAgainstSchemaTask.java b/buildSrc/src/main/java/org/opensearch/gradle/precommit/ValidateJsonAgainstSchemaTask.java index 72019f87d1ae6..ddf42fbe64d4a 100644 --- a/buildSrc/src/main/java/org/opensearch/gradle/precommit/ValidateJsonAgainstSchemaTask.java +++ b/buildSrc/src/main/java/org/opensearch/gradle/precommit/ValidateJsonAgainstSchemaTask.java @@ -114,7 +114,7 @@ public void validate(InputChanges inputChanges) throws IOException { File file = fileChange.getFile(); if (file.isDirectory() == false) { // validate all files and hold on to errors for a complete report if there are failures - getLogger().debug("Validating JSON [{}]", file.getName()); + getLogger().debug("Validating JSON [{}]", file.getName()); try { Set validationMessages = jsonSchema.validate(mapper.readTree(file)); maybeLogAndCollectError(validationMessages, errors, file); From 61bf7666a08708572c9f85f0a52aae55f3afc128 Mon Sep 17 00:00:00 2001 From: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> Date: Wed, 7 Jun 2023 15:02:10 -0400 Subject: [PATCH 07/16] Update buildSrc/src/main/java/org/opensearch/gradle/precommit/ValidateJsonAgainstSchemaTask.java Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> --- .../gradle/precommit/ValidateJsonAgainstSchemaTask.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/buildSrc/src/main/java/org/opensearch/gradle/precommit/ValidateJsonAgainstSchemaTask.java b/buildSrc/src/main/java/org/opensearch/gradle/precommit/ValidateJsonAgainstSchemaTask.java index ddf42fbe64d4a..1ecc712ab27ca 100644 --- a/buildSrc/src/main/java/org/opensearch/gradle/precommit/ValidateJsonAgainstSchemaTask.java +++ b/buildSrc/src/main/java/org/opensearch/gradle/precommit/ValidateJsonAgainstSchemaTask.java @@ -144,7 +144,7 @@ public void validate(InputChanges inputChanges) throws IOException { } } - private void maybeLogAndCollectError(Set messages, Map> errors, File file) throws IOException { + private void maybeLogAndCollectError(Set messages, Map> errors, File file) { for (ValidationMessage message : messages) { getLogger().error("[validate JSON][ERROR][{}][{}]", file.getName(), message.toString()); errors.computeIfAbsent(file, k -> new LinkedHashSet<>()) From 3518b8151be8d1366bafdbeabf3d3412de384080 Mon Sep 17 00:00:00 2001 From: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> Date: Wed, 7 Jun 2023 15:02:52 -0400 Subject: [PATCH 08/16] Update rest-api-spec/src/main/resources/rest-api-spec/api/bulk.json Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> --- .../src/main/resources/rest-api-spec/api/bulk.json | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/rest-api-spec/src/main/resources/rest-api-spec/api/bulk.json b/rest-api-spec/src/main/resources/rest-api-spec/api/bulk.json index 978d3740524a9..257bf74b7c04a 100644 --- a/rest-api-spec/src/main/resources/rest-api-spec/api/bulk.json +++ b/rest-api-spec/src/main/resources/rest-api-spec/api/bulk.json @@ -36,7 +36,11 @@ }, "refresh":{ "type":"enum", - "options": ["true", "false", "wait_for"], + "options":[ + "true", + "false", + "wait_for" + ], "description":"If `true` then refresh the affected shards to make this operation visible to search, if `wait_for` then wait for a refresh to make this operation visible to search, if `false` (the default) then do nothing with refreshes." }, "routing":{ From dcbbc8c4881e57869c20b6bca9c63a8bb08b7743 Mon Sep 17 00:00:00 2001 From: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> Date: Wed, 7 Jun 2023 15:03:06 -0400 Subject: [PATCH 09/16] Update rest-api-spec/src/main/resources/rest-api-spec/api/bulk.json Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> --- rest-api-spec/src/main/resources/rest-api-spec/api/bulk.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rest-api-spec/src/main/resources/rest-api-spec/api/bulk.json b/rest-api-spec/src/main/resources/rest-api-spec/api/bulk.json index 257bf74b7c04a..bb066cd131480 100644 --- a/rest-api-spec/src/main/resources/rest-api-spec/api/bulk.json +++ b/rest-api-spec/src/main/resources/rest-api-spec/api/bulk.json @@ -36,7 +36,7 @@ }, "refresh":{ "type":"enum", - "options":[ + "options":[ "true", "false", "wait_for" From 9d9823d98eee31ad93717cc9c6e23953cdd7e4ad Mon Sep 17 00:00:00 2001 From: Andriy Redko Date: Tue, 6 Jun 2023 17:22:53 -0400 Subject: [PATCH 10/16] Update Apache Lucene to 9.7.0-snapshot-204acc3 (#7887) Signed-off-by: Andriy Redko --- buildSrc/version.properties | 2 +- libs/core/build.gradle | 24 +++++++++++++------ ...ucene-core-9.7.0-snapshot-204acc3.jar.sha1 | 1 + ...ucene-core-9.7.0-snapshot-4d1ed9e.jar.sha1 | 1 - ...xpressions-9.7.0-snapshot-204acc3.jar.sha1 | 1 + ...xpressions-9.7.0-snapshot-4d1ed9e.jar.sha1 | 1 - ...alysis-icu-9.7.0-snapshot-204acc3.jar.sha1 | 1 + ...alysis-icu-9.7.0-snapshot-4d1ed9e.jar.sha1 | 1 - ...s-kuromoji-9.7.0-snapshot-204acc3.jar.sha1 | 1 + ...s-kuromoji-9.7.0-snapshot-4d1ed9e.jar.sha1 | 1 - ...lysis-nori-9.7.0-snapshot-204acc3.jar.sha1 | 1 + ...lysis-nori-9.7.0-snapshot-4d1ed9e.jar.sha1 | 1 - ...s-phonetic-9.7.0-snapshot-204acc3.jar.sha1 | 1 + ...s-phonetic-9.7.0-snapshot-4d1ed9e.jar.sha1 | 1 - ...is-smartcn-9.7.0-snapshot-204acc3.jar.sha1 | 1 + ...is-smartcn-9.7.0-snapshot-4d1ed9e.jar.sha1 | 1 - ...is-stempel-9.7.0-snapshot-204acc3.jar.sha1 | 1 + ...is-stempel-9.7.0-snapshot-4d1ed9e.jar.sha1 | 1 - ...morfologik-9.7.0-snapshot-204acc3.jar.sha1 | 1 + ...morfologik-9.7.0-snapshot-4d1ed9e.jar.sha1 | 1 - server/build.gradle | 10 ++++++++ ...sis-common-9.7.0-snapshot-204acc3.jar.sha1 | 1 + ...sis-common-9.7.0-snapshot-4d1ed9e.jar.sha1 | 1 - ...ard-codecs-9.7.0-snapshot-204acc3.jar.sha1 | 1 + ...ard-codecs-9.7.0-snapshot-4d1ed9e.jar.sha1 | 1 - ...ucene-core-9.7.0-snapshot-204acc3.jar.sha1 | 1 + ...ucene-core-9.7.0-snapshot-4d1ed9e.jar.sha1 | 1 - ...e-grouping-9.7.0-snapshot-204acc3.jar.sha1 | 1 + ...e-grouping-9.7.0-snapshot-4d1ed9e.jar.sha1 | 1 - ...ighlighter-9.7.0-snapshot-204acc3.jar.sha1 | 1 + ...ighlighter-9.7.0-snapshot-4d1ed9e.jar.sha1 | 1 - ...ucene-join-9.7.0-snapshot-204acc3.jar.sha1 | 1 + ...ucene-join-9.7.0-snapshot-4d1ed9e.jar.sha1 | 1 - ...ene-memory-9.7.0-snapshot-204acc3.jar.sha1 | 1 + ...ene-memory-9.7.0-snapshot-4d1ed9e.jar.sha1 | 1 - ...ucene-misc-9.7.0-snapshot-204acc3.jar.sha1 | 1 + ...ucene-misc-9.7.0-snapshot-4d1ed9e.jar.sha1 | 1 - ...ne-queries-9.7.0-snapshot-204acc3.jar.sha1 | 1 + ...ne-queries-9.7.0-snapshot-4d1ed9e.jar.sha1 | 1 - ...ueryparser-9.7.0-snapshot-204acc3.jar.sha1 | 1 + ...ueryparser-9.7.0-snapshot-4d1ed9e.jar.sha1 | 1 - ...ne-sandbox-9.7.0-snapshot-204acc3.jar.sha1 | 1 + ...ne-sandbox-9.7.0-snapshot-4d1ed9e.jar.sha1 | 1 - ...ial-extras-9.7.0-snapshot-204acc3.jar.sha1 | 1 + ...ial-extras-9.7.0-snapshot-4d1ed9e.jar.sha1 | 1 - ...-spatial3d-9.7.0-snapshot-204acc3.jar.sha1 | 1 + ...-spatial3d-9.7.0-snapshot-4d1ed9e.jar.sha1 | 1 - ...ne-suggest-9.7.0-snapshot-204acc3.jar.sha1 | 1 + ...ne-suggest-9.7.0-snapshot-4d1ed9e.jar.sha1 | 1 - .../opensearch/index/search/MatchQuery.java | 1 + .../index/search/MultiMatchQuery.java | 1 + .../analysis/AnalysisFactoryTestCase.java | 2 ++ 52 files changed, 55 insertions(+), 31 deletions(-) create mode 100644 libs/core/licenses/lucene-core-9.7.0-snapshot-204acc3.jar.sha1 delete mode 100644 libs/core/licenses/lucene-core-9.7.0-snapshot-4d1ed9e.jar.sha1 create mode 100644 modules/lang-expression/licenses/lucene-expressions-9.7.0-snapshot-204acc3.jar.sha1 delete mode 100644 modules/lang-expression/licenses/lucene-expressions-9.7.0-snapshot-4d1ed9e.jar.sha1 create mode 100644 plugins/analysis-icu/licenses/lucene-analysis-icu-9.7.0-snapshot-204acc3.jar.sha1 delete mode 100644 plugins/analysis-icu/licenses/lucene-analysis-icu-9.7.0-snapshot-4d1ed9e.jar.sha1 create mode 100644 plugins/analysis-kuromoji/licenses/lucene-analysis-kuromoji-9.7.0-snapshot-204acc3.jar.sha1 delete mode 100644 plugins/analysis-kuromoji/licenses/lucene-analysis-kuromoji-9.7.0-snapshot-4d1ed9e.jar.sha1 create mode 100644 plugins/analysis-nori/licenses/lucene-analysis-nori-9.7.0-snapshot-204acc3.jar.sha1 delete mode 100644 plugins/analysis-nori/licenses/lucene-analysis-nori-9.7.0-snapshot-4d1ed9e.jar.sha1 create mode 100644 plugins/analysis-phonetic/licenses/lucene-analysis-phonetic-9.7.0-snapshot-204acc3.jar.sha1 delete mode 100644 plugins/analysis-phonetic/licenses/lucene-analysis-phonetic-9.7.0-snapshot-4d1ed9e.jar.sha1 create mode 100644 plugins/analysis-smartcn/licenses/lucene-analysis-smartcn-9.7.0-snapshot-204acc3.jar.sha1 delete mode 100644 plugins/analysis-smartcn/licenses/lucene-analysis-smartcn-9.7.0-snapshot-4d1ed9e.jar.sha1 create mode 100644 plugins/analysis-stempel/licenses/lucene-analysis-stempel-9.7.0-snapshot-204acc3.jar.sha1 delete mode 100644 plugins/analysis-stempel/licenses/lucene-analysis-stempel-9.7.0-snapshot-4d1ed9e.jar.sha1 create mode 100644 plugins/analysis-ukrainian/licenses/lucene-analysis-morfologik-9.7.0-snapshot-204acc3.jar.sha1 delete mode 100644 plugins/analysis-ukrainian/licenses/lucene-analysis-morfologik-9.7.0-snapshot-4d1ed9e.jar.sha1 create mode 100644 server/licenses/lucene-analysis-common-9.7.0-snapshot-204acc3.jar.sha1 delete mode 100644 server/licenses/lucene-analysis-common-9.7.0-snapshot-4d1ed9e.jar.sha1 create mode 100644 server/licenses/lucene-backward-codecs-9.7.0-snapshot-204acc3.jar.sha1 delete mode 100644 server/licenses/lucene-backward-codecs-9.7.0-snapshot-4d1ed9e.jar.sha1 create mode 100644 server/licenses/lucene-core-9.7.0-snapshot-204acc3.jar.sha1 delete mode 100644 server/licenses/lucene-core-9.7.0-snapshot-4d1ed9e.jar.sha1 create mode 100644 server/licenses/lucene-grouping-9.7.0-snapshot-204acc3.jar.sha1 delete mode 100644 server/licenses/lucene-grouping-9.7.0-snapshot-4d1ed9e.jar.sha1 create mode 100644 server/licenses/lucene-highlighter-9.7.0-snapshot-204acc3.jar.sha1 delete mode 100644 server/licenses/lucene-highlighter-9.7.0-snapshot-4d1ed9e.jar.sha1 create mode 100644 server/licenses/lucene-join-9.7.0-snapshot-204acc3.jar.sha1 delete mode 100644 server/licenses/lucene-join-9.7.0-snapshot-4d1ed9e.jar.sha1 create mode 100644 server/licenses/lucene-memory-9.7.0-snapshot-204acc3.jar.sha1 delete mode 100644 server/licenses/lucene-memory-9.7.0-snapshot-4d1ed9e.jar.sha1 create mode 100644 server/licenses/lucene-misc-9.7.0-snapshot-204acc3.jar.sha1 delete mode 100644 server/licenses/lucene-misc-9.7.0-snapshot-4d1ed9e.jar.sha1 create mode 100644 server/licenses/lucene-queries-9.7.0-snapshot-204acc3.jar.sha1 delete mode 100644 server/licenses/lucene-queries-9.7.0-snapshot-4d1ed9e.jar.sha1 create mode 100644 server/licenses/lucene-queryparser-9.7.0-snapshot-204acc3.jar.sha1 delete mode 100644 server/licenses/lucene-queryparser-9.7.0-snapshot-4d1ed9e.jar.sha1 create mode 100644 server/licenses/lucene-sandbox-9.7.0-snapshot-204acc3.jar.sha1 delete mode 100644 server/licenses/lucene-sandbox-9.7.0-snapshot-4d1ed9e.jar.sha1 create mode 100644 server/licenses/lucene-spatial-extras-9.7.0-snapshot-204acc3.jar.sha1 delete mode 100644 server/licenses/lucene-spatial-extras-9.7.0-snapshot-4d1ed9e.jar.sha1 create mode 100644 server/licenses/lucene-spatial3d-9.7.0-snapshot-204acc3.jar.sha1 delete mode 100644 server/licenses/lucene-spatial3d-9.7.0-snapshot-4d1ed9e.jar.sha1 create mode 100644 server/licenses/lucene-suggest-9.7.0-snapshot-204acc3.jar.sha1 delete mode 100644 server/licenses/lucene-suggest-9.7.0-snapshot-4d1ed9e.jar.sha1 diff --git a/buildSrc/version.properties b/buildSrc/version.properties index 9b54122bf91ea..f62f2217971f7 100644 --- a/buildSrc/version.properties +++ b/buildSrc/version.properties @@ -1,5 +1,5 @@ opensearch = 3.0.0 -lucene = 9.7.0-snapshot-4d1ed9e +lucene = 9.7.0-snapshot-204acc3 bundled_jdk_vendor = adoptium bundled_jdk = 20.0.1+9 diff --git a/libs/core/build.gradle b/libs/core/build.gradle index 696ea11f5be50..6c6fc745b5942 100644 --- a/libs/core/build.gradle +++ b/libs/core/build.gradle @@ -103,13 +103,23 @@ tasks.named('forbiddenApisMain').configure { tasks.named("thirdPartyAudit").configure { ignoreMissingClasses( // from log4j - 'org.osgi.framework.Bundle', - 'org.osgi.framework.BundleActivator', - 'org.osgi.framework.BundleContext', - 'org.osgi.framework.BundleEvent', - 'org.osgi.framework.SynchronousBundleListener', - 'org.osgi.framework.wiring.BundleWire', - 'org.osgi.framework.wiring.BundleWiring' + *[ + 'org.osgi.framework.Bundle', + 'org.osgi.framework.BundleActivator', + 'org.osgi.framework.BundleContext', + 'org.osgi.framework.BundleEvent', + 'org.osgi.framework.SynchronousBundleListener', + 'org.osgi.framework.wiring.BundleWire', + 'org.osgi.framework.wiring.BundleWiring', + ] + (BuildParams.runtimeJavaVersion < JavaVersion.VERSION_20) ? [] : [ + 'jdk.incubator.vector.ByteVector', + 'jdk.incubator.vector.FloatVector', + 'jdk.incubator.vector.IntVector', + 'jdk.incubator.vector.ShortVector', + 'jdk.incubator.vector.Vector', + 'jdk.incubator.vector.VectorOperators', + 'jdk.incubator.vector.VectorSpecies' + ] ) } diff --git a/libs/core/licenses/lucene-core-9.7.0-snapshot-204acc3.jar.sha1 b/libs/core/licenses/lucene-core-9.7.0-snapshot-204acc3.jar.sha1 new file mode 100644 index 0000000000000..2afe66c03cf22 --- /dev/null +++ b/libs/core/licenses/lucene-core-9.7.0-snapshot-204acc3.jar.sha1 @@ -0,0 +1 @@ +3aa698cf90f074cbf24acfd7feaaad84c5a6f829 \ No newline at end of file diff --git a/libs/core/licenses/lucene-core-9.7.0-snapshot-4d1ed9e.jar.sha1 b/libs/core/licenses/lucene-core-9.7.0-snapshot-4d1ed9e.jar.sha1 deleted file mode 100644 index 12114423683c2..0000000000000 --- a/libs/core/licenses/lucene-core-9.7.0-snapshot-4d1ed9e.jar.sha1 +++ /dev/null @@ -1 +0,0 @@ -8e87d93868271da754a7c22517b3dbbcf84fc7f9 \ No newline at end of file diff --git a/modules/lang-expression/licenses/lucene-expressions-9.7.0-snapshot-204acc3.jar.sha1 b/modules/lang-expression/licenses/lucene-expressions-9.7.0-snapshot-204acc3.jar.sha1 new file mode 100644 index 0000000000000..f70c327ab5f6b --- /dev/null +++ b/modules/lang-expression/licenses/lucene-expressions-9.7.0-snapshot-204acc3.jar.sha1 @@ -0,0 +1 @@ +c31671978777d97c026b13decfbef1d7eeed4410 \ No newline at end of file diff --git a/modules/lang-expression/licenses/lucene-expressions-9.7.0-snapshot-4d1ed9e.jar.sha1 b/modules/lang-expression/licenses/lucene-expressions-9.7.0-snapshot-4d1ed9e.jar.sha1 deleted file mode 100644 index 805927269dcdc..0000000000000 --- a/modules/lang-expression/licenses/lucene-expressions-9.7.0-snapshot-4d1ed9e.jar.sha1 +++ /dev/null @@ -1 +0,0 @@ -441c6341e75f06b834c992084f21556681963ea6 \ No newline at end of file diff --git a/plugins/analysis-icu/licenses/lucene-analysis-icu-9.7.0-snapshot-204acc3.jar.sha1 b/plugins/analysis-icu/licenses/lucene-analysis-icu-9.7.0-snapshot-204acc3.jar.sha1 new file mode 100644 index 0000000000000..b5c59c8f5ed8c --- /dev/null +++ b/plugins/analysis-icu/licenses/lucene-analysis-icu-9.7.0-snapshot-204acc3.jar.sha1 @@ -0,0 +1 @@ +e31f7c161ad2b7652b1ac9100b4fa67d52be86eb \ No newline at end of file diff --git a/plugins/analysis-icu/licenses/lucene-analysis-icu-9.7.0-snapshot-4d1ed9e.jar.sha1 b/plugins/analysis-icu/licenses/lucene-analysis-icu-9.7.0-snapshot-4d1ed9e.jar.sha1 deleted file mode 100644 index 36207cb3ba7de..0000000000000 --- a/plugins/analysis-icu/licenses/lucene-analysis-icu-9.7.0-snapshot-4d1ed9e.jar.sha1 +++ /dev/null @@ -1 +0,0 @@ -5da04f8baf0d0961eee28fc1ca0dde93f58bd3e5 \ No newline at end of file diff --git a/plugins/analysis-kuromoji/licenses/lucene-analysis-kuromoji-9.7.0-snapshot-204acc3.jar.sha1 b/plugins/analysis-kuromoji/licenses/lucene-analysis-kuromoji-9.7.0-snapshot-204acc3.jar.sha1 new file mode 100644 index 0000000000000..e3f1d7afdc72a --- /dev/null +++ b/plugins/analysis-kuromoji/licenses/lucene-analysis-kuromoji-9.7.0-snapshot-204acc3.jar.sha1 @@ -0,0 +1 @@ +ed13aaa5843c76ba8bb9fcd1cac6de8855ea7f1c \ No newline at end of file diff --git a/plugins/analysis-kuromoji/licenses/lucene-analysis-kuromoji-9.7.0-snapshot-4d1ed9e.jar.sha1 b/plugins/analysis-kuromoji/licenses/lucene-analysis-kuromoji-9.7.0-snapshot-4d1ed9e.jar.sha1 deleted file mode 100644 index 3d7e75eca8126..0000000000000 --- a/plugins/analysis-kuromoji/licenses/lucene-analysis-kuromoji-9.7.0-snapshot-4d1ed9e.jar.sha1 +++ /dev/null @@ -1 +0,0 @@ -c5307fbb67bd0901e07d2987d4c3e33dc7d9dd62 \ No newline at end of file diff --git a/plugins/analysis-nori/licenses/lucene-analysis-nori-9.7.0-snapshot-204acc3.jar.sha1 b/plugins/analysis-nori/licenses/lucene-analysis-nori-9.7.0-snapshot-204acc3.jar.sha1 new file mode 100644 index 0000000000000..12210d41b4e6f --- /dev/null +++ b/plugins/analysis-nori/licenses/lucene-analysis-nori-9.7.0-snapshot-204acc3.jar.sha1 @@ -0,0 +1 @@ +f8673a2b86d3868e9ef2745d1d3fa9ad5a63e84f \ No newline at end of file diff --git a/plugins/analysis-nori/licenses/lucene-analysis-nori-9.7.0-snapshot-4d1ed9e.jar.sha1 b/plugins/analysis-nori/licenses/lucene-analysis-nori-9.7.0-snapshot-4d1ed9e.jar.sha1 deleted file mode 100644 index 82f1d60f1970a..0000000000000 --- a/plugins/analysis-nori/licenses/lucene-analysis-nori-9.7.0-snapshot-4d1ed9e.jar.sha1 +++ /dev/null @@ -1 +0,0 @@ -668dd1d3506840f6a258efda8a4f5e0ae0f069eb \ No newline at end of file diff --git a/plugins/analysis-phonetic/licenses/lucene-analysis-phonetic-9.7.0-snapshot-204acc3.jar.sha1 b/plugins/analysis-phonetic/licenses/lucene-analysis-phonetic-9.7.0-snapshot-204acc3.jar.sha1 new file mode 100644 index 0000000000000..307d90ca834b6 --- /dev/null +++ b/plugins/analysis-phonetic/licenses/lucene-analysis-phonetic-9.7.0-snapshot-204acc3.jar.sha1 @@ -0,0 +1 @@ +f7a585d5f62033f19a5ec79b66a74cc24c9e9d10 \ No newline at end of file diff --git a/plugins/analysis-phonetic/licenses/lucene-analysis-phonetic-9.7.0-snapshot-4d1ed9e.jar.sha1 b/plugins/analysis-phonetic/licenses/lucene-analysis-phonetic-9.7.0-snapshot-4d1ed9e.jar.sha1 deleted file mode 100644 index 96e07339bbebe..0000000000000 --- a/plugins/analysis-phonetic/licenses/lucene-analysis-phonetic-9.7.0-snapshot-4d1ed9e.jar.sha1 +++ /dev/null @@ -1 +0,0 @@ -df49e950d6463df63546b0989b46a717424b48c0 \ No newline at end of file diff --git a/plugins/analysis-smartcn/licenses/lucene-analysis-smartcn-9.7.0-snapshot-204acc3.jar.sha1 b/plugins/analysis-smartcn/licenses/lucene-analysis-smartcn-9.7.0-snapshot-204acc3.jar.sha1 new file mode 100644 index 0000000000000..b4e595a07ffda --- /dev/null +++ b/plugins/analysis-smartcn/licenses/lucene-analysis-smartcn-9.7.0-snapshot-204acc3.jar.sha1 @@ -0,0 +1 @@ +40ba622c3aa91a0a8f2747617fef8f59bf212345 \ No newline at end of file diff --git a/plugins/analysis-smartcn/licenses/lucene-analysis-smartcn-9.7.0-snapshot-4d1ed9e.jar.sha1 b/plugins/analysis-smartcn/licenses/lucene-analysis-smartcn-9.7.0-snapshot-4d1ed9e.jar.sha1 deleted file mode 100644 index 62fbf8fd39c73..0000000000000 --- a/plugins/analysis-smartcn/licenses/lucene-analysis-smartcn-9.7.0-snapshot-4d1ed9e.jar.sha1 +++ /dev/null @@ -1 +0,0 @@ -06abcf12423dc8f115f2bb4c269b6ba2202245c2 \ No newline at end of file diff --git a/plugins/analysis-stempel/licenses/lucene-analysis-stempel-9.7.0-snapshot-204acc3.jar.sha1 b/plugins/analysis-stempel/licenses/lucene-analysis-stempel-9.7.0-snapshot-204acc3.jar.sha1 new file mode 100644 index 0000000000000..9b6dcc21465fa --- /dev/null +++ b/plugins/analysis-stempel/licenses/lucene-analysis-stempel-9.7.0-snapshot-204acc3.jar.sha1 @@ -0,0 +1 @@ +e08432a3db0bde29f425812896bd695c5477b05f \ No newline at end of file diff --git a/plugins/analysis-stempel/licenses/lucene-analysis-stempel-9.7.0-snapshot-4d1ed9e.jar.sha1 b/plugins/analysis-stempel/licenses/lucene-analysis-stempel-9.7.0-snapshot-4d1ed9e.jar.sha1 deleted file mode 100644 index b038010aa271f..0000000000000 --- a/plugins/analysis-stempel/licenses/lucene-analysis-stempel-9.7.0-snapshot-4d1ed9e.jar.sha1 +++ /dev/null @@ -1 +0,0 @@ -0c2f7b681c54ff2f174a894c0bcc26030d1644c2 \ No newline at end of file diff --git a/plugins/analysis-ukrainian/licenses/lucene-analysis-morfologik-9.7.0-snapshot-204acc3.jar.sha1 b/plugins/analysis-ukrainian/licenses/lucene-analysis-morfologik-9.7.0-snapshot-204acc3.jar.sha1 new file mode 100644 index 0000000000000..b70b4e8d6c35b --- /dev/null +++ b/plugins/analysis-ukrainian/licenses/lucene-analysis-morfologik-9.7.0-snapshot-204acc3.jar.sha1 @@ -0,0 +1 @@ +a44008e83286060f0265a32f9a9b586e86ec03b1 \ No newline at end of file diff --git a/plugins/analysis-ukrainian/licenses/lucene-analysis-morfologik-9.7.0-snapshot-4d1ed9e.jar.sha1 b/plugins/analysis-ukrainian/licenses/lucene-analysis-morfologik-9.7.0-snapshot-4d1ed9e.jar.sha1 deleted file mode 100644 index 8a60f1eb31e08..0000000000000 --- a/plugins/analysis-ukrainian/licenses/lucene-analysis-morfologik-9.7.0-snapshot-4d1ed9e.jar.sha1 +++ /dev/null @@ -1 +0,0 @@ -ac785becbaf914830deeb950626b850cd89efadf \ No newline at end of file diff --git a/server/build.gradle b/server/build.gradle index 7e75368323cf2..a77fdc9062c5a 100644 --- a/server/build.gradle +++ b/server/build.gradle @@ -244,6 +244,7 @@ tasks.named("processResources").configure { tasks.named("thirdPartyAudit").configure { ignoreMissingClasses( + *[ // from com.fasterxml.jackson.dataformat.yaml.YAMLMapper (jackson-dataformat-yaml) 'com.fasterxml.jackson.databind.ObjectMapper', @@ -337,6 +338,15 @@ tasks.named("thirdPartyAudit").configure { 'com.google.common.geometry.S2Point', 'com.google.common.geometry.S2$Metric', 'com.google.common.geometry.S2LatLng' + ] + (BuildParams.runtimeJavaVersion < JavaVersion.VERSION_20) ? [] : [ + 'jdk.incubator.vector.ByteVector', + 'jdk.incubator.vector.FloatVector', + 'jdk.incubator.vector.IntVector', + 'jdk.incubator.vector.ShortVector', + 'jdk.incubator.vector.Vector', + 'jdk.incubator.vector.VectorOperators', + 'jdk.incubator.vector.VectorSpecies' + ] ) ignoreViolations( 'com.google.protobuf.MessageSchema', diff --git a/server/licenses/lucene-analysis-common-9.7.0-snapshot-204acc3.jar.sha1 b/server/licenses/lucene-analysis-common-9.7.0-snapshot-204acc3.jar.sha1 new file mode 100644 index 0000000000000..a770070d97c2d --- /dev/null +++ b/server/licenses/lucene-analysis-common-9.7.0-snapshot-204acc3.jar.sha1 @@ -0,0 +1 @@ +5aa5989b931c68eee90b22bca3f1f280a7d5c1ee \ No newline at end of file diff --git a/server/licenses/lucene-analysis-common-9.7.0-snapshot-4d1ed9e.jar.sha1 b/server/licenses/lucene-analysis-common-9.7.0-snapshot-4d1ed9e.jar.sha1 deleted file mode 100644 index 412b87be774e0..0000000000000 --- a/server/licenses/lucene-analysis-common-9.7.0-snapshot-4d1ed9e.jar.sha1 +++ /dev/null @@ -1 +0,0 @@ -d478df2acb5b224038a2380c15eec26a77bbce8b \ No newline at end of file diff --git a/server/licenses/lucene-backward-codecs-9.7.0-snapshot-204acc3.jar.sha1 b/server/licenses/lucene-backward-codecs-9.7.0-snapshot-204acc3.jar.sha1 new file mode 100644 index 0000000000000..c35bc03c19097 --- /dev/null +++ b/server/licenses/lucene-backward-codecs-9.7.0-snapshot-204acc3.jar.sha1 @@ -0,0 +1 @@ +2326fb4762cd14eff6bea770d5c85a848e7a2482 \ No newline at end of file diff --git a/server/licenses/lucene-backward-codecs-9.7.0-snapshot-4d1ed9e.jar.sha1 b/server/licenses/lucene-backward-codecs-9.7.0-snapshot-4d1ed9e.jar.sha1 deleted file mode 100644 index 690c3d6b5955d..0000000000000 --- a/server/licenses/lucene-backward-codecs-9.7.0-snapshot-4d1ed9e.jar.sha1 +++ /dev/null @@ -1 +0,0 @@ -28c618f5e814cf05af94032f340d7cb79ea7dec2 \ No newline at end of file diff --git a/server/licenses/lucene-core-9.7.0-snapshot-204acc3.jar.sha1 b/server/licenses/lucene-core-9.7.0-snapshot-204acc3.jar.sha1 new file mode 100644 index 0000000000000..2afe66c03cf22 --- /dev/null +++ b/server/licenses/lucene-core-9.7.0-snapshot-204acc3.jar.sha1 @@ -0,0 +1 @@ +3aa698cf90f074cbf24acfd7feaaad84c5a6f829 \ No newline at end of file diff --git a/server/licenses/lucene-core-9.7.0-snapshot-4d1ed9e.jar.sha1 b/server/licenses/lucene-core-9.7.0-snapshot-4d1ed9e.jar.sha1 deleted file mode 100644 index 12114423683c2..0000000000000 --- a/server/licenses/lucene-core-9.7.0-snapshot-4d1ed9e.jar.sha1 +++ /dev/null @@ -1 +0,0 @@ -8e87d93868271da754a7c22517b3dbbcf84fc7f9 \ No newline at end of file diff --git a/server/licenses/lucene-grouping-9.7.0-snapshot-204acc3.jar.sha1 b/server/licenses/lucene-grouping-9.7.0-snapshot-204acc3.jar.sha1 new file mode 100644 index 0000000000000..c7aaff3a0b184 --- /dev/null +++ b/server/licenses/lucene-grouping-9.7.0-snapshot-204acc3.jar.sha1 @@ -0,0 +1 @@ +b3251c58b21c5c205c63bbe618c0d3c1393908e1 \ No newline at end of file diff --git a/server/licenses/lucene-grouping-9.7.0-snapshot-4d1ed9e.jar.sha1 b/server/licenses/lucene-grouping-9.7.0-snapshot-4d1ed9e.jar.sha1 deleted file mode 100644 index 63b85475e130b..0000000000000 --- a/server/licenses/lucene-grouping-9.7.0-snapshot-4d1ed9e.jar.sha1 +++ /dev/null @@ -1 +0,0 @@ -b1b0674d8afa4ad78c1eccfb0b73064b5826631d \ No newline at end of file diff --git a/server/licenses/lucene-highlighter-9.7.0-snapshot-204acc3.jar.sha1 b/server/licenses/lucene-highlighter-9.7.0-snapshot-204acc3.jar.sha1 new file mode 100644 index 0000000000000..6f9bfe7604f7f --- /dev/null +++ b/server/licenses/lucene-highlighter-9.7.0-snapshot-204acc3.jar.sha1 @@ -0,0 +1 @@ +2313a0755a5774eb5dfcdc116dc15bd9367c8cc1 \ No newline at end of file diff --git a/server/licenses/lucene-highlighter-9.7.0-snapshot-4d1ed9e.jar.sha1 b/server/licenses/lucene-highlighter-9.7.0-snapshot-4d1ed9e.jar.sha1 deleted file mode 100644 index c69ef9d8ad17f..0000000000000 --- a/server/licenses/lucene-highlighter-9.7.0-snapshot-4d1ed9e.jar.sha1 +++ /dev/null @@ -1 +0,0 @@ -d376f4bad9ba6b9a2cc3fb4201cccfd7465cb54c \ No newline at end of file diff --git a/server/licenses/lucene-join-9.7.0-snapshot-204acc3.jar.sha1 b/server/licenses/lucene-join-9.7.0-snapshot-204acc3.jar.sha1 new file mode 100644 index 0000000000000..00b5c33be3568 --- /dev/null +++ b/server/licenses/lucene-join-9.7.0-snapshot-204acc3.jar.sha1 @@ -0,0 +1 @@ +25fddcaf47d5614f48dc264f79151eb87990abd9 \ No newline at end of file diff --git a/server/licenses/lucene-join-9.7.0-snapshot-4d1ed9e.jar.sha1 b/server/licenses/lucene-join-9.7.0-snapshot-4d1ed9e.jar.sha1 deleted file mode 100644 index 0b3f3bf35d4d6..0000000000000 --- a/server/licenses/lucene-join-9.7.0-snapshot-4d1ed9e.jar.sha1 +++ /dev/null @@ -1 +0,0 @@ -757e3bdda4a34fb4a5ded85d34df5ea00dc4e310 \ No newline at end of file diff --git a/server/licenses/lucene-memory-9.7.0-snapshot-204acc3.jar.sha1 b/server/licenses/lucene-memory-9.7.0-snapshot-204acc3.jar.sha1 new file mode 100644 index 0000000000000..cf7eced4686aa --- /dev/null +++ b/server/licenses/lucene-memory-9.7.0-snapshot-204acc3.jar.sha1 @@ -0,0 +1 @@ +7c2aad4c1baa818a94bd787615e70824e6ae3a9d \ No newline at end of file diff --git a/server/licenses/lucene-memory-9.7.0-snapshot-4d1ed9e.jar.sha1 b/server/licenses/lucene-memory-9.7.0-snapshot-4d1ed9e.jar.sha1 deleted file mode 100644 index 87965865e9f52..0000000000000 --- a/server/licenses/lucene-memory-9.7.0-snapshot-4d1ed9e.jar.sha1 +++ /dev/null @@ -1 +0,0 @@ -daaf3d5424e6e474e860a7db92af4f490c627457 \ No newline at end of file diff --git a/server/licenses/lucene-misc-9.7.0-snapshot-204acc3.jar.sha1 b/server/licenses/lucene-misc-9.7.0-snapshot-204acc3.jar.sha1 new file mode 100644 index 0000000000000..b5ac7eb1f988b --- /dev/null +++ b/server/licenses/lucene-misc-9.7.0-snapshot-204acc3.jar.sha1 @@ -0,0 +1 @@ +56211acb2a31b1967568eba9953db502a53f9010 \ No newline at end of file diff --git a/server/licenses/lucene-misc-9.7.0-snapshot-4d1ed9e.jar.sha1 b/server/licenses/lucene-misc-9.7.0-snapshot-4d1ed9e.jar.sha1 deleted file mode 100644 index ecbc173a2e2fc..0000000000000 --- a/server/licenses/lucene-misc-9.7.0-snapshot-4d1ed9e.jar.sha1 +++ /dev/null @@ -1 +0,0 @@ -c3c52d69784be7787e239631cc7b64f3e32ded4f \ No newline at end of file diff --git a/server/licenses/lucene-queries-9.7.0-snapshot-204acc3.jar.sha1 b/server/licenses/lucene-queries-9.7.0-snapshot-204acc3.jar.sha1 new file mode 100644 index 0000000000000..2aa628ebc9573 --- /dev/null +++ b/server/licenses/lucene-queries-9.7.0-snapshot-204acc3.jar.sha1 @@ -0,0 +1 @@ +c45b1847150e2e2e6302177b41faf95df2ca4fbf \ No newline at end of file diff --git a/server/licenses/lucene-queries-9.7.0-snapshot-4d1ed9e.jar.sha1 b/server/licenses/lucene-queries-9.7.0-snapshot-4d1ed9e.jar.sha1 deleted file mode 100644 index 42b204fa512f3..0000000000000 --- a/server/licenses/lucene-queries-9.7.0-snapshot-4d1ed9e.jar.sha1 +++ /dev/null @@ -1 +0,0 @@ -8f10d4ff130928351a8c03f9c2d10188d0267d74 \ No newline at end of file diff --git a/server/licenses/lucene-queryparser-9.7.0-snapshot-204acc3.jar.sha1 b/server/licenses/lucene-queryparser-9.7.0-snapshot-204acc3.jar.sha1 new file mode 100644 index 0000000000000..5e7b5d9da9c0d --- /dev/null +++ b/server/licenses/lucene-queryparser-9.7.0-snapshot-204acc3.jar.sha1 @@ -0,0 +1 @@ +d40fe6400f21564ccb50b6716ec42d8dcd526b9d \ No newline at end of file diff --git a/server/licenses/lucene-queryparser-9.7.0-snapshot-4d1ed9e.jar.sha1 b/server/licenses/lucene-queryparser-9.7.0-snapshot-4d1ed9e.jar.sha1 deleted file mode 100644 index fb00bb91e49e7..0000000000000 --- a/server/licenses/lucene-queryparser-9.7.0-snapshot-4d1ed9e.jar.sha1 +++ /dev/null @@ -1 +0,0 @@ -22bc9f5ca57b5b3c35b10075716ca6b2ab1e2f4c \ No newline at end of file diff --git a/server/licenses/lucene-sandbox-9.7.0-snapshot-204acc3.jar.sha1 b/server/licenses/lucene-sandbox-9.7.0-snapshot-204acc3.jar.sha1 new file mode 100644 index 0000000000000..e45e9b929836b --- /dev/null +++ b/server/licenses/lucene-sandbox-9.7.0-snapshot-204acc3.jar.sha1 @@ -0,0 +1 @@ +e9347773d0b269768c58dd0b438bdc4b450f9185 \ No newline at end of file diff --git a/server/licenses/lucene-sandbox-9.7.0-snapshot-4d1ed9e.jar.sha1 b/server/licenses/lucene-sandbox-9.7.0-snapshot-4d1ed9e.jar.sha1 deleted file mode 100644 index 815fec16716c4..0000000000000 --- a/server/licenses/lucene-sandbox-9.7.0-snapshot-4d1ed9e.jar.sha1 +++ /dev/null @@ -1 +0,0 @@ -e88b9b8adb0e206a20482f47340cb65794d23330 \ No newline at end of file diff --git a/server/licenses/lucene-spatial-extras-9.7.0-snapshot-204acc3.jar.sha1 b/server/licenses/lucene-spatial-extras-9.7.0-snapshot-204acc3.jar.sha1 new file mode 100644 index 0000000000000..a697cb9fa2dff --- /dev/null +++ b/server/licenses/lucene-spatial-extras-9.7.0-snapshot-204acc3.jar.sha1 @@ -0,0 +1 @@ +7055502c9a77b9a93d1b96c99397fd765dd7891f \ No newline at end of file diff --git a/server/licenses/lucene-spatial-extras-9.7.0-snapshot-4d1ed9e.jar.sha1 b/server/licenses/lucene-spatial-extras-9.7.0-snapshot-4d1ed9e.jar.sha1 deleted file mode 100644 index 6035ad3a79715..0000000000000 --- a/server/licenses/lucene-spatial-extras-9.7.0-snapshot-4d1ed9e.jar.sha1 +++ /dev/null @@ -1 +0,0 @@ -85b32ddf62395219fe5bd7d5d9f70e81c9a41d34 \ No newline at end of file diff --git a/server/licenses/lucene-spatial3d-9.7.0-snapshot-204acc3.jar.sha1 b/server/licenses/lucene-spatial3d-9.7.0-snapshot-204acc3.jar.sha1 new file mode 100644 index 0000000000000..3b96e448add5b --- /dev/null +++ b/server/licenses/lucene-spatial3d-9.7.0-snapshot-204acc3.jar.sha1 @@ -0,0 +1 @@ +a2c5a7be5887dacf89c251936c3f3388ca20d28e \ No newline at end of file diff --git a/server/licenses/lucene-spatial3d-9.7.0-snapshot-4d1ed9e.jar.sha1 b/server/licenses/lucene-spatial3d-9.7.0-snapshot-4d1ed9e.jar.sha1 deleted file mode 100644 index 4ad2e869c9eb8..0000000000000 --- a/server/licenses/lucene-spatial3d-9.7.0-snapshot-4d1ed9e.jar.sha1 +++ /dev/null @@ -1 +0,0 @@ -387188af30ac63d8e970b2f3043003b01ed001ca \ No newline at end of file diff --git a/server/licenses/lucene-suggest-9.7.0-snapshot-204acc3.jar.sha1 b/server/licenses/lucene-suggest-9.7.0-snapshot-204acc3.jar.sha1 new file mode 100644 index 0000000000000..0460861900897 --- /dev/null +++ b/server/licenses/lucene-suggest-9.7.0-snapshot-204acc3.jar.sha1 @@ -0,0 +1 @@ +1a076a8db84fa85634cc4894ee4fc78b5e69525c \ No newline at end of file diff --git a/server/licenses/lucene-suggest-9.7.0-snapshot-4d1ed9e.jar.sha1 b/server/licenses/lucene-suggest-9.7.0-snapshot-4d1ed9e.jar.sha1 deleted file mode 100644 index 73f330f5a96ce..0000000000000 --- a/server/licenses/lucene-suggest-9.7.0-snapshot-4d1ed9e.jar.sha1 +++ /dev/null @@ -1 +0,0 @@ -c9aa971ca1beaa1fe407417587343c0a356b9ad2 \ No newline at end of file diff --git a/server/src/main/java/org/opensearch/index/search/MatchQuery.java b/server/src/main/java/org/opensearch/index/search/MatchQuery.java index 91a4b456bfa2a..2c8b091fef8ef 100644 --- a/server/src/main/java/org/opensearch/index/search/MatchQuery.java +++ b/server/src/main/java/org/opensearch/index/search/MatchQuery.java @@ -56,6 +56,7 @@ import org.apache.lucene.search.Query; import org.apache.lucene.search.TermQuery; import org.apache.lucene.util.QueryBuilder; +import org.apache.lucene.util.TermAndBoost; import org.apache.lucene.util.graph.GraphTokenStreamFiniteStrings; import org.opensearch.OpenSearchException; import org.opensearch.common.io.stream.StreamInput; diff --git a/server/src/main/java/org/opensearch/index/search/MultiMatchQuery.java b/server/src/main/java/org/opensearch/index/search/MultiMatchQuery.java index 241f05af2c512..554d9714409b7 100644 --- a/server/src/main/java/org/opensearch/index/search/MultiMatchQuery.java +++ b/server/src/main/java/org/opensearch/index/search/MultiMatchQuery.java @@ -42,6 +42,7 @@ import org.apache.lucene.search.Query; import org.apache.lucene.search.TermQuery; import org.apache.lucene.util.BytesRef; +import org.apache.lucene.util.TermAndBoost; import org.opensearch.common.lucene.search.Queries; import org.opensearch.index.mapper.MappedFieldType; import org.opensearch.index.query.AbstractQueryBuilder; diff --git a/test/framework/src/main/java/org/opensearch/indices/analysis/AnalysisFactoryTestCase.java b/test/framework/src/main/java/org/opensearch/indices/analysis/AnalysisFactoryTestCase.java index 27f3312626e48..b93cb64e32cfe 100644 --- a/test/framework/src/main/java/org/opensearch/indices/analysis/AnalysisFactoryTestCase.java +++ b/test/framework/src/main/java/org/opensearch/indices/analysis/AnalysisFactoryTestCase.java @@ -222,6 +222,8 @@ public abstract class AnalysisFactoryTestCase extends OpenSearchTestCase { // LUCENE-10352 .put("daitchmokotoffsoundex", Void.class) .put("persianstem", Void.class) + // https://github.com/apache/lucene/pull/12169 + .put("word2vecsynonym", Void.class) .immutableMap(); static final Map> KNOWN_CHARFILTERS = new MapBuilder>() From 95885071678e2f80769c329ae7bae04eba4eab7a Mon Sep 17 00:00:00 2001 From: Andriy Redko Date: Tue, 6 Jun 2023 17:37:58 -0400 Subject: [PATCH 11/16] Add bwc version 2.8.1 (#7942) Signed-off-by: Andriy Redko Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> --- .ci/bwcVersions | 1 + libs/core/src/main/java/org/opensearch/Version.java | 1 + 2 files changed, 2 insertions(+) diff --git a/.ci/bwcVersions b/.ci/bwcVersions index 09e8acf54090e..7fc76dc8c6c04 100644 --- a/.ci/bwcVersions +++ b/.ci/bwcVersions @@ -19,4 +19,5 @@ BWC_VERSION: - "2.7.0" - "2.7.1" - "2.8.0" + - "2.8.1" - "2.9.0" diff --git a/libs/core/src/main/java/org/opensearch/Version.java b/libs/core/src/main/java/org/opensearch/Version.java index 79ccef4063d9b..c399f7f3e1b1c 100644 --- a/libs/core/src/main/java/org/opensearch/Version.java +++ b/libs/core/src/main/java/org/opensearch/Version.java @@ -88,6 +88,7 @@ public class Version implements Comparable, ToXContentFragment { public static final Version V_2_7_0 = new Version(2070099, org.apache.lucene.util.Version.LUCENE_9_5_0); public static final Version V_2_7_1 = new Version(2070199, org.apache.lucene.util.Version.LUCENE_9_5_0); public static final Version V_2_8_0 = new Version(2080099, org.apache.lucene.util.Version.LUCENE_9_6_0); + public static final Version V_2_8_1 = new Version(2080199, org.apache.lucene.util.Version.LUCENE_9_6_0); public static final Version V_2_9_0 = new Version(2090099, org.apache.lucene.util.Version.LUCENE_9_6_0); public static final Version V_3_0_0 = new Version(3000099, org.apache.lucene.util.Version.LUCENE_9_7_0); public static final Version CURRENT = V_3_0_0; From e0c82fa736fff5762594b946f8b456e4dc599199 Mon Sep 17 00:00:00 2001 From: Andrew Ross Date: Tue, 6 Jun 2023 19:13:47 -0500 Subject: [PATCH 12/16] Fix changelog entries (#7946) Move entry for #7321 to correct section and fix the malformed entry from #7452. Signed-off-by: Andrew Ross --- CHANGELOG.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f74ebba9b4ba0..f2e8087cf0d37 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -61,7 +61,6 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), - Add ZSTD compression for snapshotting ([#2996](https://github.com/opensearch-project/OpenSearch/pull/2996)) - Change `com.amazonaws.sdk.ec2MetadataServiceEndpointOverride` to `aws.ec2MetadataServiceEndpoint` ([7372](https://github.com/opensearch-project/OpenSearch/pull/7372/)) - Change `com.amazonaws.sdk.stsEndpointOverride` to `aws.stsEndpointOverride` ([7372](https://github.com/opensearch-project/OpenSearch/pull/7372/)) -- Compress and cache cluster state during validate join request ([#7321](https://github.com/opensearch-project/OpenSearch/pull/7321)) ### Deprecated @@ -91,7 +90,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), ### Security ## [Unreleased 2.x] -### Add TokenManager Interface ([#7452](https://github.com/opensearch-project/OpenSearch/pull/7452)) +### Added +- Add TokenManager Interface ([#7452](https://github.com/opensearch-project/OpenSearch/pull/7452)) ### Dependencies - Bump `jackson` from 2.15.1 to 2.15.2 ([#7897](https://github.com/opensearch-project/OpenSearch/pull/7897)) @@ -100,6 +100,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), ### Changed - Replace jboss-annotations-api_1.2_spec with jakarta.annotation-api ([#7836](https://github.com/opensearch-project/OpenSearch/pull/7836)) - Add min, max, average and thread info to resource stats in tasks API ([#7673](https://github.com/opensearch-project/OpenSearch/pull/7673)) +- Compress and cache cluster state during validate join request ([#7321](https://github.com/opensearch-project/OpenSearch/pull/7321)) ### Deprecated From 8e792a9f0f6b572c388cdb5c3252507c581efcdc Mon Sep 17 00:00:00 2001 From: Ankit Kala Date: Wed, 7 Jun 2023 22:26:04 +0530 Subject: [PATCH 13/16] SegRep with Remote: Add Remote store as a segment replication source (#7653) * SegRep with Remote: Add Remote store as a segment replication source Signed-off-by: Ankit Kala * Fix Gradle check Signed-off-by: Ankit Kala * Retrying Gradle check Signed-off-by: Ankit Kala --------- Signed-off-by: Ankit Kala --- CHANGELOG.md | 1 + .../replication/SegmentReplicationIT.java | 16 ++ .../SegmentReplicationRemoteStoreIT.java | 2 +- .../SegmentReplicationUsingRemoteStoreIT.java | 66 ++++++++ .../index/engine/InternalEngine.java | 10 +- .../opensearch/index/shard/IndexShard.java | 65 ++++---- .../shard/RemoteStoreRefreshListener.java | 7 +- .../opensearch/index/shard/StoreRecovery.java | 2 +- .../index/store/RemoteDirectory.java | 11 +- .../store/RemoteSegmentStoreDirectory.java | 7 +- .../org/opensearch/index/store/Store.java | 3 +- .../recovery/PeerRecoveryTargetService.java | 2 +- .../PrimaryShardReplicationSource.java | 4 +- .../RemoteStoreReplicationSource.java | 99 ++++++++++++ .../replication/SegmentReplicationSource.java | 6 +- .../SegmentReplicationSourceFactory.java | 18 ++- .../replication/SegmentReplicationTarget.java | 10 +- .../index/engine/InternalEngineTests.java | 2 + .../RemoteStoreRefreshListenerTests.java | 4 +- .../SegmentReplicationIndexShardTests.java | 8 +- ...tReplicationWithRemoteIndexShardTests.java | 43 +++++ .../PrimaryShardReplicationSourceTests.java | 7 +- .../RemoteStoreReplicationSourceTests.java | 150 ++++++++++++++++++ .../SegmentReplicationTargetServiceTests.java | 3 +- .../SegmentReplicationTargetTests.java | 14 +- .../replication/TestReplicationSource.java | 4 +- .../index/shard/IndexShardTestCase.java | 2 +- 27 files changed, 485 insertions(+), 81 deletions(-) create mode 100644 server/src/internalClusterTest/java/org/opensearch/remotestore/SegmentReplicationUsingRemoteStoreIT.java create mode 100644 server/src/main/java/org/opensearch/indices/replication/RemoteStoreReplicationSource.java create mode 100644 server/src/test/java/org/opensearch/index/shard/SegmentReplicationWithRemoteIndexShardTests.java create mode 100644 server/src/test/java/org/opensearch/indices/replication/RemoteStoreReplicationSourceTests.java diff --git a/CHANGELOG.md b/CHANGELOG.md index f2e8087cf0d37..f84c90d2b65bb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -92,6 +92,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), ## [Unreleased 2.x] ### Added - Add TokenManager Interface ([#7452](https://github.com/opensearch-project/OpenSearch/pull/7452)) +- Add Remote store as a segment replication source ([#7653](https://github.com/opensearch-project/OpenSearch/pull/7653)) ### Dependencies - Bump `jackson` from 2.15.1 to 2.15.2 ([#7897](https://github.com/opensearch-project/OpenSearch/pull/7897)) diff --git a/server/src/internalClusterTest/java/org/opensearch/indices/replication/SegmentReplicationIT.java b/server/src/internalClusterTest/java/org/opensearch/indices/replication/SegmentReplicationIT.java index c4420922d1dcb..08d1878c2df27 100644 --- a/server/src/internalClusterTest/java/org/opensearch/indices/replication/SegmentReplicationIT.java +++ b/server/src/internalClusterTest/java/org/opensearch/indices/replication/SegmentReplicationIT.java @@ -47,6 +47,7 @@ import org.opensearch.common.settings.Settings; import org.opensearch.common.unit.TimeValue; import org.opensearch.core.common.lease.Releasable; +import org.opensearch.common.util.FeatureFlags; import org.opensearch.index.IndexModule; import org.opensearch.index.SegmentReplicationPerGroupStats; import org.opensearch.index.SegmentReplicationPressureService; @@ -580,6 +581,7 @@ public void testDeleteOperations() throws Exception { * from xlog. */ public void testReplicationPostDeleteAndForceMerge() throws Exception { + assumeFalse("Skipping the test with Remote store as its flaky.", segmentReplicationWithRemoteEnabled()); final String primary = internalCluster().startNode(); createIndex(INDEX_NAME); final String replica = internalCluster().startNode(); @@ -785,6 +787,10 @@ public void testReplicaHasDiffFilesThanPrimary() throws Exception { } public void testPressureServiceStats() throws Exception { + assumeFalse( + "Skipping the test as pressure service is not compatible with SegRep and Remote store yet.", + segmentReplicationWithRemoteEnabled() + ); final String primaryNode = internalCluster().startNode(); createIndex(INDEX_NAME); final String replicaNode = internalCluster().startNode(); @@ -874,6 +880,7 @@ public void testPressureServiceStats() throws Exception { * @throws Exception when issue is encountered */ public void testScrollCreatedOnReplica() throws Exception { + assumeFalse("Skipping the test with Remote store as its flaky.", segmentReplicationWithRemoteEnabled()); // create the cluster with one primary node containing primary shard and replica node containing replica shard final String primary = internalCluster().startNode(); createIndex(INDEX_NAME); @@ -963,6 +970,11 @@ public void testScrollCreatedOnReplica() throws Exception { * @throws Exception when issue is encountered */ public void testScrollWithOngoingSegmentReplication() throws Exception { + assumeFalse( + "Skipping the test as its not compatible with segment replication with remote store yet.", + segmentReplicationWithRemoteEnabled() + ); + // create the cluster with one primary node containing primary shard and replica node containing replica shard final String primary = internalCluster().startNode(); prepareCreate( @@ -1249,4 +1261,8 @@ public void testPrimaryReceivesDocsDuringReplicaRecovery() throws Exception { waitForSearchableDocs(2, nodes); } + private boolean segmentReplicationWithRemoteEnabled() { + return IndexMetadata.INDEX_REMOTE_STORE_ENABLED_SETTING.get(indexSettings()).booleanValue() + && "true".equalsIgnoreCase(featureFlagSettings().get(FeatureFlags.SEGMENT_REPLICATION_EXPERIMENTAL)); + } } diff --git a/server/src/internalClusterTest/java/org/opensearch/remotestore/SegmentReplicationRemoteStoreIT.java b/server/src/internalClusterTest/java/org/opensearch/remotestore/SegmentReplicationRemoteStoreIT.java index 055f997fbe197..ab0c0cc3aec77 100644 --- a/server/src/internalClusterTest/java/org/opensearch/remotestore/SegmentReplicationRemoteStoreIT.java +++ b/server/src/internalClusterTest/java/org/opensearch/remotestore/SegmentReplicationRemoteStoreIT.java @@ -30,7 +30,7 @@ @OpenSearchIntegTestCase.ClusterScope(scope = OpenSearchIntegTestCase.Scope.TEST, numDataNodes = 0) public class SegmentReplicationRemoteStoreIT extends SegmentReplicationIT { - private static final String REPOSITORY_NAME = "test-remore-store-repo"; + private static final String REPOSITORY_NAME = "test-remote-store-repo"; @Override public Settings indexSettings() { diff --git a/server/src/internalClusterTest/java/org/opensearch/remotestore/SegmentReplicationUsingRemoteStoreIT.java b/server/src/internalClusterTest/java/org/opensearch/remotestore/SegmentReplicationUsingRemoteStoreIT.java new file mode 100644 index 0000000000000..ad6e358cb9da1 --- /dev/null +++ b/server/src/internalClusterTest/java/org/opensearch/remotestore/SegmentReplicationUsingRemoteStoreIT.java @@ -0,0 +1,66 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * + * The OpenSearch Contributors require contributions made to + * this file be licensed under the Apache-2.0 license or a + * compatible open source license. + */ + +package org.opensearch.remotestore; + +import org.junit.After; +import org.junit.Before; +import org.opensearch.cluster.metadata.IndexMetadata; +import org.opensearch.common.settings.Settings; +import org.opensearch.common.util.FeatureFlags; +import org.opensearch.indices.replication.SegmentReplicationIT; +import org.opensearch.test.OpenSearchIntegTestCase; + +import java.nio.file.Path; + +import static org.opensearch.test.hamcrest.OpenSearchAssertions.assertAcked; + +/** + * This class runs Segment Replication Integ test suite with remote store enabled. + * Setup is similar to SegmentReplicationRemoteStoreIT but this also enables the segment replication using remote store which + * is behind SEGMENT_REPLICATION_EXPERIMENTAL flag. After this is moved out of experimental, we can combine and keep only one + * test suite for Segment and Remote store integration tests. + */ +@OpenSearchIntegTestCase.ClusterScope(scope = OpenSearchIntegTestCase.Scope.TEST, numDataNodes = 0) +public class SegmentReplicationUsingRemoteStoreIT extends SegmentReplicationIT { + + private static final String REPOSITORY_NAME = "test-remote-store-repo"; + + @Override + public Settings indexSettings() { + return Settings.builder() + .put(super.indexSettings()) + .put(IndexMetadata.SETTING_REMOTE_STORE_ENABLED, true) + .put(IndexMetadata.SETTING_REMOTE_STORE_REPOSITORY, REPOSITORY_NAME) + .put(IndexMetadata.SETTING_REMOTE_TRANSLOG_STORE_ENABLED, false) + .build(); + } + + @Override + protected Settings featureFlagSettings() { + return Settings.builder() + .put(super.featureFlagSettings()) + .put(FeatureFlags.REMOTE_STORE, "true") + .put(FeatureFlags.SEGMENT_REPLICATION_EXPERIMENTAL, "true") + .build(); + } + + @Before + public void setup() { + internalCluster().startClusterManagerOnlyNode(); + Path absolutePath = randomRepoPath().toAbsolutePath(); + assertAcked( + clusterAdmin().preparePutRepository(REPOSITORY_NAME).setType("fs").setSettings(Settings.builder().put("location", absolutePath)) + ); + } + + @After + public void teardown() { + assertAcked(clusterAdmin().prepareDeleteRepository(REPOSITORY_NAME)); + } +} diff --git a/server/src/main/java/org/opensearch/index/engine/InternalEngine.java b/server/src/main/java/org/opensearch/index/engine/InternalEngine.java index e777a9ef724a9..63afc6585a99d 100644 --- a/server/src/main/java/org/opensearch/index/engine/InternalEngine.java +++ b/server/src/main/java/org/opensearch/index/engine/InternalEngine.java @@ -2760,6 +2760,13 @@ public final long lastRefreshedCheckpoint() { return lastRefreshedCheckpointListener.refreshedCheckpoint.get(); } + /** + * Returns the current local checkpoint getting refreshed internally. + */ + public final long currentOngoingRefreshCheckpoint() { + return lastRefreshedCheckpointListener.pendingCheckpoint; + } + private final Object refreshIfNeededMutex = new Object(); /** @@ -2777,10 +2784,11 @@ protected final void refreshIfNeeded(String source, long requestingSeqNo) { private final class LastRefreshedCheckpointListener implements ReferenceManager.RefreshListener { final AtomicLong refreshedCheckpoint; - private long pendingCheckpoint; + volatile long pendingCheckpoint; LastRefreshedCheckpointListener(long initialLocalCheckpoint) { this.refreshedCheckpoint = new AtomicLong(initialLocalCheckpoint); + this.pendingCheckpoint = initialLocalCheckpoint; } @Override diff --git a/server/src/main/java/org/opensearch/index/shard/IndexShard.java b/server/src/main/java/org/opensearch/index/shard/IndexShard.java index d6ab3a764abf2..ecc46e720fdaf 100644 --- a/server/src/main/java/org/opensearch/index/shard/IndexShard.java +++ b/server/src/main/java/org/opensearch/index/shard/IndexShard.java @@ -2231,7 +2231,7 @@ private void innerOpenEngineAndTranslog(LongSupplier globalCheckpointSupplier) t assert currentEngineReference.get() == null : "engine is running"; verifyNotClosed(); if (indexSettings.isRemoteStoreEnabled()) { - syncSegmentsFromRemoteSegmentStore(false); + syncSegmentsFromRemoteSegmentStore(false, true, true); } if (indexSettings.isRemoteTranslogStoreEnabled() && shardRouting.primary()) { syncRemoteTranslogAndUpdateGlobalCheckpoint(); @@ -4404,7 +4404,7 @@ public void close() throws IOException { }; IOUtils.close(currentEngineReference.getAndSet(readOnlyEngine)); if (indexSettings.isRemoteStoreEnabled()) { - syncSegmentsFromRemoteSegmentStore(false); + syncSegmentsFromRemoteSegmentStore(false, true, true); } if (indexSettings.isRemoteTranslogStoreEnabled() && shardRouting.primary()) { syncRemoteTranslogAndUpdateGlobalCheckpoint(); @@ -4454,23 +4454,15 @@ public void syncTranslogFilesFromRemoteTranslog() throws IOException { RemoteFsTranslog.download(repository, shardId, getThreadPool(), shardPath().resolveTranslog()); } - /** - * Downloads segments from remote segment store. This method will download segments till - * last refresh checkpoint. - * @param overrideLocal flag to override local segment files with those in remote store - * @throws IOException if exception occurs while reading segments from remote store - */ - public void syncSegmentsFromRemoteSegmentStore(boolean overrideLocal) throws IOException { - syncSegmentsFromRemoteSegmentStore(overrideLocal, true); - } - /** * Downloads segments from remote segment store. * @param overrideLocal flag to override local segment files with those in remote store * @param refreshLevelSegmentSync last refresh checkpoint is used if true, commit checkpoint otherwise + * @param shouldCommit if the shard requires committing the changes after sync from remote. * @throws IOException if exception occurs while reading segments from remote store */ - public void syncSegmentsFromRemoteSegmentStore(boolean overrideLocal, boolean refreshLevelSegmentSync) throws IOException { + public void syncSegmentsFromRemoteSegmentStore(boolean overrideLocal, boolean refreshLevelSegmentSync, boolean shouldCommit) + throws IOException { assert indexSettings.isRemoteStoreEnabled(); logger.info("Downloading segments from remote segment store"); assert remoteStore.directory() instanceof FilterDirectory : "Store.directory is not an instance of FilterDirectory"; @@ -4529,29 +4521,34 @@ public void syncSegmentsFromRemoteSegmentStore(boolean overrideLocal, boolean re indexInput, remoteSegmentMetadata.getGeneration() ); - long processedLocalCheckpoint = Long.parseLong(infosSnapshot.getUserData().get(LOCAL_CHECKPOINT_KEY)); - // Following code block makes sure to use SegmentInfosSnapshot in the remote store if generation differs - // with local filesystem. If local filesystem already has segments_N+2 and infosSnapshot has generation N, - // after commit, there would be 2 files that would be created segments_N+1 and segments_N+2. With the - // policy of preserving only the latest commit, we will delete segments_N+1 which in fact is the part of the latest - // commit. - Optional localMaxSegmentInfos = localSegmentFiles.stream() - .filter(file -> file.startsWith(IndexFileNames.SEGMENTS)) - .max(Comparator.comparingLong(SegmentInfos::generationFromSegmentsFileName)); - if (localMaxSegmentInfos.isPresent() - && infosSnapshot.getGeneration() < SegmentInfos.generationFromSegmentsFileName(localMaxSegmentInfos.get()) - 1) { - // If remote translog is not enabled, local translog will be created with different UUID. - // This fails in Store.trimUnsafeCommits() as translog UUID of checkpoint and SegmentInfos needs - // to be same. Following code block make sure to have the same UUID. - if (indexSettings.isRemoteTranslogStoreEnabled() == false) { - SegmentInfos localSegmentInfos = store.readLastCommittedSegmentsInfo(); - Map userData = new HashMap<>(infosSnapshot.getUserData()); - userData.put(TRANSLOG_UUID_KEY, localSegmentInfos.userData.get(TRANSLOG_UUID_KEY)); - infosSnapshot.setUserData(userData, false); + if (shouldCommit) { + long processedLocalCheckpoint = Long.parseLong(infosSnapshot.getUserData().get(LOCAL_CHECKPOINT_KEY)); + // Following code block makes sure to use SegmentInfosSnapshot in the remote store if generation differs + // with local filesystem. If local filesystem already has segments_N+2 and infosSnapshot has generation N, + // after commit, there would be 2 files that would be created segments_N+1 and segments_N+2. With the + // policy of preserving only the latest commit, we will delete segments_N+1 which in fact is the part of the latest + // commit. + Optional localMaxSegmentInfos = localSegmentFiles.stream() + .filter(file -> file.startsWith(IndexFileNames.SEGMENTS)) + .max(Comparator.comparingLong(SegmentInfos::generationFromSegmentsFileName)); + if (localMaxSegmentInfos.isPresent() + && infosSnapshot.getGeneration() < SegmentInfos.generationFromSegmentsFileName(localMaxSegmentInfos.get()) + - 1) { + // If remote translog is not enabled, local translog will be created with different UUID. + // This fails in Store.trimUnsafeCommits() as translog UUID of checkpoint and SegmentInfos needs + // to be same. Following code block make sure to have the same UUID. + if (indexSettings.isRemoteTranslogStoreEnabled() == false) { + SegmentInfos localSegmentInfos = store.readLastCommittedSegmentsInfo(); + Map userData = new HashMap<>(infosSnapshot.getUserData()); + userData.put(TRANSLOG_UUID_KEY, localSegmentInfos.userData.get(TRANSLOG_UUID_KEY)); + infosSnapshot.setUserData(userData, false); + } + storeDirectory.deleteFile(localMaxSegmentInfos.get()); } - storeDirectory.deleteFile(localMaxSegmentInfos.get()); + store.commitSegmentInfos(infosSnapshot, processedLocalCheckpoint, processedLocalCheckpoint); + } else { + finalizeReplication(infosSnapshot); } - store.commitSegmentInfos(infosSnapshot, processedLocalCheckpoint, processedLocalCheckpoint); } } } catch (IOException e) { diff --git a/server/src/main/java/org/opensearch/index/shard/RemoteStoreRefreshListener.java b/server/src/main/java/org/opensearch/index/shard/RemoteStoreRefreshListener.java index 8f71a9f3ff658..b24a78971b71f 100644 --- a/server/src/main/java/org/opensearch/index/shard/RemoteStoreRefreshListener.java +++ b/server/src/main/java/org/opensearch/index/shard/RemoteStoreRefreshListener.java @@ -334,12 +334,11 @@ private boolean isRefreshAfterCommit() throws IOException { } void uploadMetadata(Collection localSegmentsPostRefresh, SegmentInfos segmentInfos) throws IOException { - final long maxSeqNoFromSegmentInfos = indexShard.getEngine().getMaxSeqNoFromSegmentInfos(segmentInfos); - + final long maxSeqNo = ((InternalEngine) indexShard.getEngine()).currentOngoingRefreshCheckpoint(); SegmentInfos segmentInfosSnapshot = segmentInfos.clone(); Map userData = segmentInfosSnapshot.getUserData(); - userData.put(LOCAL_CHECKPOINT_KEY, String.valueOf(maxSeqNoFromSegmentInfos)); - userData.put(SequenceNumbers.MAX_SEQ_NO, Long.toString(maxSeqNoFromSegmentInfos)); + userData.put(LOCAL_CHECKPOINT_KEY, String.valueOf(maxSeqNo)); + userData.put(SequenceNumbers.MAX_SEQ_NO, Long.toString(maxSeqNo)); segmentInfosSnapshot.setUserData(userData, false); remoteDirectory.uploadMetadata( diff --git a/server/src/main/java/org/opensearch/index/shard/StoreRecovery.java b/server/src/main/java/org/opensearch/index/shard/StoreRecovery.java index 9393a5ac38ac2..02397bc356539 100644 --- a/server/src/main/java/org/opensearch/index/shard/StoreRecovery.java +++ b/server/src/main/java/org/opensearch/index/shard/StoreRecovery.java @@ -453,7 +453,7 @@ private void recoverFromRemoteStore(IndexShard indexShard) throws IndexShardReco remoteStore.incRef(); try { // Download segments from remote segment store - indexShard.syncSegmentsFromRemoteSegmentStore(true); + indexShard.syncSegmentsFromRemoteSegmentStore(true, true, true); if (store.directory().listAll().length == 0) { store.createEmpty(indexShard.indexSettings().getIndexVersionCreated().luceneVersion); diff --git a/server/src/main/java/org/opensearch/index/store/RemoteDirectory.java b/server/src/main/java/org/opensearch/index/store/RemoteDirectory.java index bb2ff47cf0072..5192fd49b91f6 100644 --- a/server/src/main/java/org/opensearch/index/store/RemoteDirectory.java +++ b/server/src/main/java/org/opensearch/index/store/RemoteDirectory.java @@ -18,6 +18,7 @@ import java.io.FileNotFoundException; import java.io.IOException; +import java.io.InputStream; import java.nio.file.NoSuchFileException; import java.util.Collection; import java.util.Collections; @@ -100,7 +101,15 @@ public IndexOutput createOutput(String name, IOContext context) { */ @Override public IndexInput openInput(String name, IOContext context) throws IOException { - return new RemoteIndexInput(name, blobContainer.readBlob(name), fileLength(name)); + InputStream inputStream = null; + try { + inputStream = blobContainer.readBlob(name); + return new RemoteIndexInput(name, inputStream, fileLength(name)); + } catch (Exception e) { + // Incase the RemoteIndexInput creation fails, close the input stream to avoid file handler leak. + if (inputStream != null) inputStream.close(); + throw e; + } } /** diff --git a/server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java b/server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java index 6549bfc6d91f4..93fa4b7eff7b7 100644 --- a/server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java +++ b/server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java @@ -137,8 +137,7 @@ public RemoteSegmentMetadata init() throws IOException { * @return Map of segment filename to uploaded filename with checksum * @throws IOException if there were any failures in reading the metadata file */ - private RemoteSegmentMetadata readLatestMetadataFile() throws IOException { - Map segmentMetadataMap = new HashMap<>(); + public RemoteSegmentMetadata readLatestMetadataFile() throws IOException { RemoteSegmentMetadata remoteSegmentMetadata = null; Collection metadataFiles = remoteMetadataDirectory.listFilesByPrefix(MetadataFilenameUtils.METADATA_PREFIX); @@ -199,6 +198,10 @@ public static UploadedSegmentMetadata fromString(String uploadedFilename) { String[] values = uploadedFilename.split(SEPARATOR); return new UploadedSegmentMetadata(values[0], values[1], values[2], Long.parseLong(values[3])); } + + public String getOriginalFilename() { + return originalFilename; + } } /** diff --git a/server/src/main/java/org/opensearch/index/store/Store.java b/server/src/main/java/org/opensearch/index/store/Store.java index dae698b1c3b46..2c0d5decebba8 100644 --- a/server/src/main/java/org/opensearch/index/store/Store.java +++ b/server/src/main/java/org/opensearch/index/store/Store.java @@ -399,7 +399,8 @@ public static RecoveryDiff segmentReplicationDiff(Map missing.add(value); } else { final StoreFileMetadata fileMetadata = target.get(value.name()); - if (fileMetadata.isSame(value)) { + // match segments using checksum + if (fileMetadata.checksum().equals(value.checksum())) { identical.add(value); } else { different.add(value); diff --git a/server/src/main/java/org/opensearch/indices/recovery/PeerRecoveryTargetService.java b/server/src/main/java/org/opensearch/indices/recovery/PeerRecoveryTargetService.java index 5fb84a165c498..a08d3182fa156 100644 --- a/server/src/main/java/org/opensearch/indices/recovery/PeerRecoveryTargetService.java +++ b/server/src/main/java/org/opensearch/indices/recovery/PeerRecoveryTargetService.java @@ -245,7 +245,7 @@ private void doRecovery(final long recoveryId, final StartRecoveryRequest preExi indexShard.prepareForIndexRecovery(); final boolean hasRemoteSegmentStore = indexShard.indexSettings().isRemoteStoreEnabled(); if (hasRemoteSegmentStore) { - indexShard.syncSegmentsFromRemoteSegmentStore(false, false); + indexShard.syncSegmentsFromRemoteSegmentStore(false, false, true); } final boolean hasRemoteTranslog = recoveryTarget.state().getPrimary() == false && indexShard.isRemoteTranslogEnabled(); final boolean hasNoTranslog = indexShard.indexSettings().isRemoteSnapshot(); diff --git a/server/src/main/java/org/opensearch/indices/replication/PrimaryShardReplicationSource.java b/server/src/main/java/org/opensearch/indices/replication/PrimaryShardReplicationSource.java index b211d81c1c76a..5455be2a69799 100644 --- a/server/src/main/java/org/opensearch/indices/replication/PrimaryShardReplicationSource.java +++ b/server/src/main/java/org/opensearch/indices/replication/PrimaryShardReplicationSource.java @@ -13,7 +13,7 @@ import org.opensearch.action.ActionListener; import org.opensearch.cluster.node.DiscoveryNode; import org.opensearch.common.io.stream.Writeable; -import org.opensearch.index.store.Store; +import org.opensearch.index.shard.IndexShard; import org.opensearch.index.store.StoreFileMetadata; import org.opensearch.indices.recovery.RecoverySettings; import org.opensearch.indices.recovery.RetryableTransportClient; @@ -79,7 +79,7 @@ public void getSegmentFiles( long replicationId, ReplicationCheckpoint checkpoint, List filesToFetch, - Store store, + IndexShard indexShard, ActionListener listener ) { final Writeable.Reader reader = GetSegmentFilesResponse::new; diff --git a/server/src/main/java/org/opensearch/indices/replication/RemoteStoreReplicationSource.java b/server/src/main/java/org/opensearch/indices/replication/RemoteStoreReplicationSource.java new file mode 100644 index 0000000000000..b8941737a7c57 --- /dev/null +++ b/server/src/main/java/org/opensearch/indices/replication/RemoteStoreReplicationSource.java @@ -0,0 +1,99 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * + * The OpenSearch Contributors require contributions made to + * this file be licensed under the Apache-2.0 license or a + * compatible open source license. + */ + +package org.opensearch.indices.replication; + +import org.apache.logging.log4j.LogManager; +import org.apache.logging.log4j.Logger; +import org.apache.lucene.store.FilterDirectory; +import org.apache.lucene.util.Version; +import org.opensearch.action.ActionListener; +import org.opensearch.index.shard.IndexShard; +import org.opensearch.index.store.RemoteSegmentStoreDirectory; +import org.opensearch.index.store.Store; +import org.opensearch.index.store.StoreFileMetadata; +import org.opensearch.indices.replication.checkpoint.ReplicationCheckpoint; + +import java.util.Collections; +import java.util.List; +import java.util.Map; +import java.util.stream.Collectors; + +/** + * Implementation of a {@link SegmentReplicationSource} where the source is remote store. + * + * @opensearch.internal + */ +public class RemoteStoreReplicationSource implements SegmentReplicationSource { + + private static final Logger logger = LogManager.getLogger(PrimaryShardReplicationSource.class); + + private final IndexShard indexShard; + + public RemoteStoreReplicationSource(IndexShard indexShard) { + this.indexShard = indexShard; + } + + @Override + public void getCheckpointMetadata( + long replicationId, + ReplicationCheckpoint checkpoint, + ActionListener listener + ) { + FilterDirectory remoteStoreDirectory = (FilterDirectory) indexShard.remoteStore().directory(); + FilterDirectory byteSizeCachingStoreDirectory = (FilterDirectory) remoteStoreDirectory.getDelegate(); + RemoteSegmentStoreDirectory remoteDirectory = (RemoteSegmentStoreDirectory) byteSizeCachingStoreDirectory.getDelegate(); + + Map metadataMap; + // TODO: Need to figure out a way to pass this information for segment metadata via remote store. + final Version version = indexShard.getSegmentInfosSnapshot().get().getCommitLuceneVersion(); + try { + metadataMap = remoteDirectory.readLatestMetadataFile() + .getMetadata() + .entrySet() + .stream() + .collect( + Collectors.toMap( + e -> e.getKey(), + e -> new StoreFileMetadata( + e.getValue().getOriginalFilename(), + e.getValue().getLength(), + Store.digestToString(Long.valueOf(e.getValue().getChecksum())), + version, + null + ) + ) + ); + // TODO: GET current checkpoint from remote store. + listener.onResponse(new CheckpointInfoResponse(checkpoint, metadataMap, null)); + } catch (Exception e) { + listener.onFailure(e); + } + } + + @Override + public void getSegmentFiles( + long replicationId, + ReplicationCheckpoint checkpoint, + List filesToFetch, + IndexShard indexShard, + ActionListener listener + ) { + try { + indexShard.syncSegmentsFromRemoteSegmentStore(false, true, false); + listener.onResponse(new GetSegmentFilesResponse(Collections.emptyList())); + } catch (Exception e) { + listener.onFailure(e); + } + } + + @Override + public String getDescription() { + return "remote store"; + } +} diff --git a/server/src/main/java/org/opensearch/indices/replication/SegmentReplicationSource.java b/server/src/main/java/org/opensearch/indices/replication/SegmentReplicationSource.java index 2fa74819fe4de..79b9b31e3d5c3 100644 --- a/server/src/main/java/org/opensearch/indices/replication/SegmentReplicationSource.java +++ b/server/src/main/java/org/opensearch/indices/replication/SegmentReplicationSource.java @@ -10,7 +10,7 @@ import org.opensearch.action.ActionListener; import org.opensearch.common.util.CancellableThreads.ExecutionCancelledException; -import org.opensearch.index.store.Store; +import org.opensearch.index.shard.IndexShard; import org.opensearch.index.store.StoreFileMetadata; import org.opensearch.indices.replication.checkpoint.ReplicationCheckpoint; @@ -38,14 +38,14 @@ public interface SegmentReplicationSource { * @param replicationId {@link long} - ID of the replication event. * @param checkpoint {@link ReplicationCheckpoint} Checkpoint to fetch metadata for. * @param filesToFetch {@link List} List of files to fetch. - * @param store {@link Store} Reference to the local store. + * @param indexShard {@link IndexShard} Reference to the IndexShard. * @param listener {@link ActionListener} Listener that completes with the list of files copied. */ void getSegmentFiles( long replicationId, ReplicationCheckpoint checkpoint, List filesToFetch, - Store store, + IndexShard indexShard, ActionListener listener ); diff --git a/server/src/main/java/org/opensearch/indices/replication/SegmentReplicationSourceFactory.java b/server/src/main/java/org/opensearch/indices/replication/SegmentReplicationSourceFactory.java index 1867fc59c5a56..238e316c3b585 100644 --- a/server/src/main/java/org/opensearch/indices/replication/SegmentReplicationSourceFactory.java +++ b/server/src/main/java/org/opensearch/indices/replication/SegmentReplicationSourceFactory.java @@ -38,13 +38,17 @@ public SegmentReplicationSourceFactory( } public SegmentReplicationSource get(IndexShard shard) { - return new PrimaryShardReplicationSource( - shard.recoveryState().getTargetNode(), - shard.routingEntry().allocationId().getId(), - transportService, - recoverySettings, - getPrimaryNode(shard.shardId()) - ); + if (shard.indexSettings().isSegRepWithRemoteEnabled()) { + return new RemoteStoreReplicationSource(shard); + } else { + return new PrimaryShardReplicationSource( + shard.recoveryState().getTargetNode(), + shard.routingEntry().allocationId().getId(), + transportService, + recoverySettings, + getPrimaryNode(shard.shardId()) + ); + } } private DiscoveryNode getPrimaryNode(ShardId shardId) { diff --git a/server/src/main/java/org/opensearch/indices/replication/SegmentReplicationTarget.java b/server/src/main/java/org/opensearch/indices/replication/SegmentReplicationTarget.java index ae722b8742e94..22c68ad46fea6 100644 --- a/server/src/main/java/org/opensearch/indices/replication/SegmentReplicationTarget.java +++ b/server/src/main/java/org/opensearch/indices/replication/SegmentReplicationTarget.java @@ -204,10 +204,18 @@ private void getFiles(CheckpointInfoResponse checkpointInfo, StepListener listener) { + // TODO: Refactor the logic so that finalize doesn't have to be invoked for remote store as source + if (source instanceof RemoteStoreReplicationSource) { + ActionListener.completeWith(listener, () -> { + state.setStage(SegmentReplicationState.Stage.FINALIZE_REPLICATION); + return null; + }); + return; + } ActionListener.completeWith(listener, () -> { cancellableThreads.checkForCancel(); state.setStage(SegmentReplicationState.Stage.FINALIZE_REPLICATION); diff --git a/server/src/test/java/org/opensearch/index/engine/InternalEngineTests.java b/server/src/test/java/org/opensearch/index/engine/InternalEngineTests.java index 91e45bd28f710..3bd8e09cab777 100644 --- a/server/src/test/java/org/opensearch/index/engine/InternalEngineTests.java +++ b/server/src/test/java/org/opensearch/index/engine/InternalEngineTests.java @@ -6637,6 +6637,7 @@ public void testLastRefreshCheckpoint() throws Exception { while (done.get() == false) { long checkPointBeforeRefresh = engine.getProcessedLocalCheckpoint(); engine.refresh("test", randomFrom(Engine.SearcherScope.values()), true); + assertThat(engine.currentOngoingRefreshCheckpoint(), greaterThanOrEqualTo(engine.lastRefreshedCheckpoint())); assertThat(engine.lastRefreshedCheckpoint(), greaterThanOrEqualTo(checkPointBeforeRefresh)); } }); @@ -6650,6 +6651,7 @@ public void testLastRefreshCheckpoint() throws Exception { thread.join(); } engine.refresh("test"); + assertThat(engine.currentOngoingRefreshCheckpoint(), greaterThanOrEqualTo(engine.lastRefreshedCheckpoint())); assertThat(engine.lastRefreshedCheckpoint(), equalTo(engine.getProcessedLocalCheckpoint())); } diff --git a/server/src/test/java/org/opensearch/index/shard/RemoteStoreRefreshListenerTests.java b/server/src/test/java/org/opensearch/index/shard/RemoteStoreRefreshListenerTests.java index 8408463a820b6..8dea8d272b4e8 100644 --- a/server/src/test/java/org/opensearch/index/shard/RemoteStoreRefreshListenerTests.java +++ b/server/src/test/java/org/opensearch/index/shard/RemoteStoreRefreshListenerTests.java @@ -421,14 +421,14 @@ private Tuple m return Tuple.tuple(refreshListener, remoteRefreshSegmentPressureService); } - private static class TestFilterDirectory extends FilterDirectory { + public static class TestFilterDirectory extends FilterDirectory { /** * Sole constructor, typically called from sub-classes. * * @param in input directory */ - protected TestFilterDirectory(Directory in) { + public TestFilterDirectory(Directory in) { super(in); } } diff --git a/server/src/test/java/org/opensearch/index/shard/SegmentReplicationIndexShardTests.java b/server/src/test/java/org/opensearch/index/shard/SegmentReplicationIndexShardTests.java index 1d0bc4cd1910b..0ee3e81678511 100644 --- a/server/src/test/java/org/opensearch/index/shard/SegmentReplicationIndexShardTests.java +++ b/server/src/test/java/org/opensearch/index/shard/SegmentReplicationIndexShardTests.java @@ -841,7 +841,7 @@ public void getSegmentFiles( long replicationId, ReplicationCheckpoint checkpoint, List filesToFetch, - Store store, + IndexShard indexShard, ActionListener listener ) { listener.onResponse(new GetSegmentFilesResponse(Collections.emptyList())); @@ -911,7 +911,7 @@ public void getSegmentFiles( long replicationId, ReplicationCheckpoint checkpoint, List filesToFetch, - Store store, + IndexShard indexShard, ActionListener listener ) { Assert.fail("Should not be reached"); @@ -951,7 +951,7 @@ public void getSegmentFiles( long replicationId, ReplicationCheckpoint checkpoint, List filesToFetch, - Store store, + IndexShard indexShard, ActionListener listener ) { // randomly resolve the listener, indicating the source has resolved. @@ -993,7 +993,7 @@ public void getSegmentFiles( long replicationId, ReplicationCheckpoint checkpoint, List filesToFetch, - Store store, + IndexShard indexShard, ActionListener listener ) {} }; diff --git a/server/src/test/java/org/opensearch/index/shard/SegmentReplicationWithRemoteIndexShardTests.java b/server/src/test/java/org/opensearch/index/shard/SegmentReplicationWithRemoteIndexShardTests.java new file mode 100644 index 0000000000000..cae15a2c53f3f --- /dev/null +++ b/server/src/test/java/org/opensearch/index/shard/SegmentReplicationWithRemoteIndexShardTests.java @@ -0,0 +1,43 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * + * The OpenSearch Contributors require contributions made to + * this file be licensed under the Apache-2.0 license or a + * compatible open source license. + */ + +package org.opensearch.index.shard; + +import org.opensearch.cluster.metadata.IndexMetadata; +import org.opensearch.common.settings.Settings; +import org.opensearch.index.engine.NRTReplicationEngineFactory; +import org.opensearch.index.replication.OpenSearchIndexLevelReplicationTestCase; +import org.opensearch.indices.replication.common.ReplicationType; + +import java.io.IOException; + +public class SegmentReplicationWithRemoteIndexShardTests extends OpenSearchIndexLevelReplicationTestCase { + private static final Settings settings = Settings.builder() + .put(IndexMetadata.SETTING_REPLICATION_TYPE, ReplicationType.SEGMENT) + .put(IndexMetadata.SETTING_REMOTE_STORE_ENABLED, true) + .put(IndexMetadata.SETTING_REMOTE_STORE_REPOSITORY, "temp-fs") + .put(IndexMetadata.SETTING_REMOTE_TRANSLOG_STORE_ENABLED, false) + .build(); + + public void testReplicaSyncingFromRemoteStore() throws IOException { + ReplicationGroup shards = createGroup(1, settings, indexMapping, new NRTReplicationEngineFactory(), createTempDir()); + final IndexShard primaryShard = shards.getPrimary(); + final IndexShard replicaShard = shards.getReplicas().get(0); + shards.startPrimary(); + shards.startAll(); + indexDoc(primaryShard, "_doc", "1"); + indexDoc(primaryShard, "_doc", "2"); + primaryShard.refresh("test"); + assertDocs(primaryShard, "1", "2"); + flushShard(primaryShard); + + replicaShard.syncSegmentsFromRemoteSegmentStore(true, true, false); + assertDocs(replicaShard, "1", "2"); + closeShards(primaryShard, replicaShard); + } +} diff --git a/server/src/test/java/org/opensearch/indices/replication/PrimaryShardReplicationSourceTests.java b/server/src/test/java/org/opensearch/indices/replication/PrimaryShardReplicationSourceTests.java index fdd707ae88195..4d273c71e7861 100644 --- a/server/src/test/java/org/opensearch/indices/replication/PrimaryShardReplicationSourceTests.java +++ b/server/src/test/java/org/opensearch/indices/replication/PrimaryShardReplicationSourceTests.java @@ -21,7 +21,6 @@ import org.opensearch.common.util.io.IOUtils; import org.opensearch.index.shard.IndexShard; import org.opensearch.index.shard.IndexShardTestCase; -import org.opensearch.index.store.Store; import org.opensearch.index.store.StoreFileMetadata; import org.opensearch.indices.recovery.RecoverySettings; import org.opensearch.indices.replication.checkpoint.ReplicationCheckpoint; @@ -123,7 +122,7 @@ public void testGetSegmentFiles() { REPLICATION_ID, checkpoint, Arrays.asList(testMetadata), - mock(Store.class), + mock(IndexShard.class), mock(ActionListener.class) ); CapturingTransport.CapturedRequest[] requestList = transport.getCapturedRequestsAndClear(); @@ -151,7 +150,7 @@ public void testTransportTimeoutForGetSegmentFilesAction() { REPLICATION_ID, checkpoint, Arrays.asList(testMetadata), - mock(Store.class), + mock(IndexShard.class), mock(ActionListener.class) ); CapturingTransport.CapturedRequest[] requestList = transport.getCapturedRequestsAndClear(); @@ -176,7 +175,7 @@ public void testGetSegmentFiles_CancelWhileRequestOpen() throws InterruptedExcep REPLICATION_ID, checkpoint, Arrays.asList(testMetadata), - mock(Store.class), + mock(IndexShard.class), new ActionListener<>() { @Override public void onResponse(GetSegmentFilesResponse getSegmentFilesResponse) { diff --git a/server/src/test/java/org/opensearch/indices/replication/RemoteStoreReplicationSourceTests.java b/server/src/test/java/org/opensearch/indices/replication/RemoteStoreReplicationSourceTests.java new file mode 100644 index 0000000000000..04f821a5fc48c --- /dev/null +++ b/server/src/test/java/org/opensearch/indices/replication/RemoteStoreReplicationSourceTests.java @@ -0,0 +1,150 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * + * The OpenSearch Contributors require contributions made to + * this file be licensed under the Apache-2.0 license or a + * compatible open source license. + */ + +package org.opensearch.indices.replication; + +import org.apache.lucene.codecs.Codec; +import org.apache.lucene.store.FilterDirectory; +import org.mockito.Mockito; +import org.opensearch.action.support.PlainActionFuture; +import org.opensearch.cluster.metadata.IndexMetadata; +import org.opensearch.common.settings.Settings; +import org.opensearch.index.engine.InternalEngineFactory; +import org.opensearch.index.replication.OpenSearchIndexLevelReplicationTestCase; +import org.opensearch.index.shard.IndexShard; +import org.opensearch.index.shard.RemoteStoreRefreshListenerTests; +import org.opensearch.index.store.RemoteSegmentStoreDirectory; +import org.opensearch.index.store.Store; +import org.opensearch.indices.replication.checkpoint.ReplicationCheckpoint; + +import java.io.IOException; +import java.util.Collections; +import java.util.concurrent.ExecutionException; +import java.util.concurrent.TimeUnit; + +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.when; + +public class RemoteStoreReplicationSourceTests extends OpenSearchIndexLevelReplicationTestCase { + + private static final long PRIMARY_TERM = 1L; + private static final long SEGMENTS_GEN = 2L; + private static final long VERSION = 4L; + private static final long REPLICATION_ID = 123L; + private RemoteStoreReplicationSource replicationSource; + private IndexShard indexShard; + + private IndexShard mockShard; + + @Override + public void setUp() throws Exception { + super.setUp(); + indexShard = newStartedShard( + true, + Settings.builder().put(IndexMetadata.SETTING_REMOTE_STORE_ENABLED, true).build(), + new InternalEngineFactory() + ); + + indexDoc(indexShard, "_doc", "1"); + indexDoc(indexShard, "_doc", "2"); + indexShard.refresh("test"); + + // mock shard + mockShard = mock(IndexShard.class); + Store store = mock(Store.class); + when(mockShard.store()).thenReturn(store); + when(store.directory()).thenReturn(indexShard.store().directory()); + Store remoteStore = mock(Store.class); + when(mockShard.remoteStore()).thenReturn(remoteStore); + RemoteSegmentStoreDirectory remoteSegmentStoreDirectory = + (RemoteSegmentStoreDirectory) ((FilterDirectory) ((FilterDirectory) indexShard.remoteStore().directory()).getDelegate()) + .getDelegate(); + FilterDirectory remoteStoreFilterDirectory = new RemoteStoreRefreshListenerTests.TestFilterDirectory( + new RemoteStoreRefreshListenerTests.TestFilterDirectory(remoteSegmentStoreDirectory) + ); + when(remoteStore.directory()).thenReturn(remoteStoreFilterDirectory); + + replicationSource = new RemoteStoreReplicationSource(mockShard); + } + + @Override + public void tearDown() throws Exception { + closeShards(indexShard); + super.tearDown(); + } + + public void testGetCheckpointMetadata() throws ExecutionException, InterruptedException { + when(mockShard.getSegmentInfosSnapshot()).thenReturn(indexShard.getSegmentInfosSnapshot()); + final ReplicationCheckpoint checkpoint = new ReplicationCheckpoint( + indexShard.shardId(), + PRIMARY_TERM, + SEGMENTS_GEN, + VERSION, + Codec.getDefault().getName() + ); + + final PlainActionFuture res = PlainActionFuture.newFuture(); + replicationSource.getCheckpointMetadata(REPLICATION_ID, checkpoint, res); + CheckpointInfoResponse response = res.get(); + assert (response.getCheckpoint().equals(checkpoint)); + assert (!response.getMetadataMap().isEmpty()); + } + + public void testGetCheckpointMetadataFailure() { + final ReplicationCheckpoint checkpoint = new ReplicationCheckpoint( + indexShard.shardId(), + PRIMARY_TERM, + SEGMENTS_GEN, + VERSION, + Codec.getDefault().getName() + ); + + when(mockShard.getSegmentInfosSnapshot()).thenThrow(new RuntimeException("test")); + + assertThrows(RuntimeException.class, () -> { + final PlainActionFuture res = PlainActionFuture.newFuture(); + replicationSource.getCheckpointMetadata(REPLICATION_ID, checkpoint, res); + res.get(); + }); + } + + public void testGetSegmentFiles() throws ExecutionException, InterruptedException { + final ReplicationCheckpoint checkpoint = new ReplicationCheckpoint( + indexShard.shardId(), + PRIMARY_TERM, + SEGMENTS_GEN, + VERSION, + Codec.getDefault().getName() + ); + + final PlainActionFuture res = PlainActionFuture.newFuture(); + replicationSource.getSegmentFiles(REPLICATION_ID, checkpoint, Collections.emptyList(), indexShard, res); + GetSegmentFilesResponse response = res.get(); + assert (response.files.isEmpty()); + assertEquals("remote store", replicationSource.getDescription()); + + } + + public void testGetSegmentFilesFailure() throws IOException { + final ReplicationCheckpoint checkpoint = new ReplicationCheckpoint( + indexShard.shardId(), + PRIMARY_TERM, + SEGMENTS_GEN, + VERSION, + Codec.getDefault().getName() + ); + Mockito.doThrow(new RuntimeException("testing")) + .when(mockShard) + .syncSegmentsFromRemoteSegmentStore(Mockito.anyBoolean(), Mockito.anyBoolean(), Mockito.anyBoolean()); + assertThrows(ExecutionException.class, () -> { + final PlainActionFuture res = PlainActionFuture.newFuture(); + replicationSource.getSegmentFiles(REPLICATION_ID, checkpoint, Collections.emptyList(), mockShard, res); + res.get(10, TimeUnit.SECONDS); + }); + } +} diff --git a/server/src/test/java/org/opensearch/indices/replication/SegmentReplicationTargetServiceTests.java b/server/src/test/java/org/opensearch/indices/replication/SegmentReplicationTargetServiceTests.java index 1d1777758972c..9c796ec05c22a 100644 --- a/server/src/test/java/org/opensearch/indices/replication/SegmentReplicationTargetServiceTests.java +++ b/server/src/test/java/org/opensearch/indices/replication/SegmentReplicationTargetServiceTests.java @@ -22,7 +22,6 @@ import org.opensearch.index.replication.TestReplicationSource; import org.opensearch.index.shard.IndexShard; import org.opensearch.index.shard.IndexShardTestCase; -import org.opensearch.index.store.Store; import org.opensearch.index.store.StoreFileMetadata; import org.opensearch.indices.IndicesService; import org.opensearch.indices.recovery.ForceSyncRequest; @@ -181,7 +180,7 @@ public void getSegmentFiles( long replicationId, ReplicationCheckpoint checkpoint, List filesToFetch, - Store store, + IndexShard indexShard, ActionListener listener ) { Assert.fail("Should not be called"); diff --git a/server/src/test/java/org/opensearch/indices/replication/SegmentReplicationTargetTests.java b/server/src/test/java/org/opensearch/indices/replication/SegmentReplicationTargetTests.java index 0e711af1afa62..ac8904527f7fb 100644 --- a/server/src/test/java/org/opensearch/indices/replication/SegmentReplicationTargetTests.java +++ b/server/src/test/java/org/opensearch/indices/replication/SegmentReplicationTargetTests.java @@ -128,7 +128,7 @@ public void getSegmentFiles( long replicationId, ReplicationCheckpoint checkpoint, List filesToFetch, - Store store, + IndexShard indexShard, ActionListener listener ) { assertEquals(1, filesToFetch.size()); @@ -179,7 +179,7 @@ public void getSegmentFiles( long replicationId, ReplicationCheckpoint checkpoint, List filesToFetch, - Store store, + IndexShard indexShard, ActionListener listener ) { listener.onResponse(new GetSegmentFilesResponse(filesToFetch)); @@ -222,7 +222,7 @@ public void getSegmentFiles( long replicationId, ReplicationCheckpoint checkpoint, List filesToFetch, - Store store, + IndexShard indexShard, ActionListener listener ) { listener.onFailure(exception); @@ -265,7 +265,7 @@ public void getSegmentFiles( long replicationId, ReplicationCheckpoint checkpoint, List filesToFetch, - Store store, + IndexShard indexShard, ActionListener listener ) { listener.onResponse(new GetSegmentFilesResponse(filesToFetch)); @@ -310,7 +310,7 @@ public void getSegmentFiles( long replicationId, ReplicationCheckpoint checkpoint, List filesToFetch, - Store store, + IndexShard indexShard, ActionListener listener ) { listener.onResponse(new GetSegmentFilesResponse(filesToFetch)); @@ -354,7 +354,7 @@ public void getSegmentFiles( long replicationId, ReplicationCheckpoint checkpoint, List filesToFetch, - Store store, + IndexShard indexShard, ActionListener listener ) { listener.onResponse(new GetSegmentFilesResponse(filesToFetch)); @@ -405,7 +405,7 @@ public void getSegmentFiles( long replicationId, ReplicationCheckpoint checkpoint, List filesToFetch, - Store store, + IndexShard indexShard, ActionListener listener ) { listener.onResponse(new GetSegmentFilesResponse(filesToFetch)); diff --git a/test/framework/src/main/java/org/opensearch/index/replication/TestReplicationSource.java b/test/framework/src/main/java/org/opensearch/index/replication/TestReplicationSource.java index a3adedcbdef86..cf4b3800069bf 100644 --- a/test/framework/src/main/java/org/opensearch/index/replication/TestReplicationSource.java +++ b/test/framework/src/main/java/org/opensearch/index/replication/TestReplicationSource.java @@ -9,7 +9,7 @@ package org.opensearch.index.replication; import org.opensearch.action.ActionListener; -import org.opensearch.index.store.Store; +import org.opensearch.index.shard.IndexShard; import org.opensearch.index.store.StoreFileMetadata; import org.opensearch.indices.replication.CheckpointInfoResponse; import org.opensearch.indices.replication.GetSegmentFilesResponse; @@ -35,7 +35,7 @@ public abstract void getSegmentFiles( long replicationId, ReplicationCheckpoint checkpoint, List filesToFetch, - Store store, + IndexShard indexShard, ActionListener listener ); diff --git a/test/framework/src/main/java/org/opensearch/index/shard/IndexShardTestCase.java b/test/framework/src/main/java/org/opensearch/index/shard/IndexShardTestCase.java index 9681c01688337..5a01fcaa1ddaf 100644 --- a/test/framework/src/main/java/org/opensearch/index/shard/IndexShardTestCase.java +++ b/test/framework/src/main/java/org/opensearch/index/shard/IndexShardTestCase.java @@ -1339,7 +1339,7 @@ public void getSegmentFiles( long replicationId, ReplicationCheckpoint checkpoint, List filesToFetch, - Store store, + IndexShard indexShard, ActionListener listener ) { try ( From ad8cab2db78987e3d7e2186567f5e4b3f735af4c Mon Sep 17 00:00:00 2001 From: Stephen Crawford Date: Wed, 7 Jun 2023 16:03:30 -0400 Subject: [PATCH 14/16] rebase Signed-off-by: Stephen Crawford --- CHANGELOG.md | 1 + buildSrc/build.gradle | 1 + 2 files changed, 2 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index f84c90d2b65bb..a619bba73f28d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -49,6 +49,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), - Bump `com.networknt:json-schema-validator` from 1.0.81 to 1.0.83 ([7933](https://github.com/opensearch-project/OpenSearch/pull/7933/)) - Bump `com.google.http-client:google-http-client:1.43.2` from 1.42.0 to 1.43.2 ([7928](https://github.com/opensearch-project/OpenSearch/pull/7928))) - Bump `com.azure:azure-storage-blob` from 12.21.1 to 12.22.2 ([7930](https://github.com/opensearch-project/OpenSearch/pull/7930)) +- Bump `com.networknt:json-schema-validator` from 1.0.81 to 1.0.83 ### Changed - [CCR] Add getHistoryOperationsFromTranslog method to fetch the history snapshot from translogs ([#3948](https://github.com/opensearch-project/OpenSearch/pull/3948)) diff --git a/buildSrc/build.gradle b/buildSrc/build.gradle index bacba8b367ddb..6cb7cb022bd6b 100644 --- a/buildSrc/build.gradle +++ b/buildSrc/build.gradle @@ -121,6 +121,7 @@ dependencies { api 'com.networknt:json-schema-validator:1.0.83' api 'org.jruby.jcodings:jcodings:1.0.58' api 'org.jruby.joni:joni:2.1.48' + api "com.fasterxml.jackson.core:jackson-databind:${props.getProperty('jackson_databind')}" testFixturesApi "junit:junit:${props.getProperty('junit')}" From a97e8ae97206e75d6f7127c6a4ef01db543c1dfb Mon Sep 17 00:00:00 2001 From: Stephen Crawford Date: Wed, 7 Jun 2023 16:08:07 -0400 Subject: [PATCH 15/16] remove empty line Signed-off-by: Stephen Crawford --- buildSrc/build.gradle | 1 - 1 file changed, 1 deletion(-) diff --git a/buildSrc/build.gradle b/buildSrc/build.gradle index 6cb7cb022bd6b..bacba8b367ddb 100644 --- a/buildSrc/build.gradle +++ b/buildSrc/build.gradle @@ -121,7 +121,6 @@ dependencies { api 'com.networknt:json-schema-validator:1.0.83' api 'org.jruby.jcodings:jcodings:1.0.58' api 'org.jruby.joni:joni:2.1.48' - api "com.fasterxml.jackson.core:jackson-databind:${props.getProperty('jackson_databind')}" testFixturesApi "junit:junit:${props.getProperty('junit')}" From 31fa1ae911b5ef8f992dc892e06a7e7c5931051a Mon Sep 17 00:00:00 2001 From: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> Date: Thu, 8 Jun 2023 09:13:40 -0400 Subject: [PATCH 16/16] Update CHANGELOG.md Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> --- CHANGELOG.md | 1 - 1 file changed, 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a619bba73f28d..f84c90d2b65bb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -49,7 +49,6 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), - Bump `com.networknt:json-schema-validator` from 1.0.81 to 1.0.83 ([7933](https://github.com/opensearch-project/OpenSearch/pull/7933/)) - Bump `com.google.http-client:google-http-client:1.43.2` from 1.42.0 to 1.43.2 ([7928](https://github.com/opensearch-project/OpenSearch/pull/7928))) - Bump `com.azure:azure-storage-blob` from 12.21.1 to 12.22.2 ([7930](https://github.com/opensearch-project/OpenSearch/pull/7930)) -- Bump `com.networknt:json-schema-validator` from 1.0.81 to 1.0.83 ### Changed - [CCR] Add getHistoryOperationsFromTranslog method to fetch the history snapshot from translogs ([#3948](https://github.com/opensearch-project/OpenSearch/pull/3948))