MapperService has to be passed in as null for EnginePlugins CodecService constructor #2177

reta · 2022-02-18T20:48:09Z

Signed-off-by: Andriy Redko andriy.redko@aiven.io

Description

Introduces CodecServiceFactory and extends the EnginePlugin with new method (the existing getCustomCodecService is marked as @Deprecated).

    default Optional<CodecServiceFactory> getCustomCodecServiceFactory() {
        return Optional.empty();
    }

The factory accepts CodecServiceConfig instance which provides more context to the CodecService creators, including MapperService is available for the particular index.

Issues Resolved

Closes #1805

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

opensearch-ci-bot · 2022-02-18T20:50:03Z

Can one of the admins verify this patch?

reta · 2022-02-18T20:50:08Z

server/src/main/java/org/opensearch/index/engine/EngineConfigFactory.java

        Optional<TranslogDeletionPolicyFactory> translogDeletionPolicyFactory = Optional.empty();
        String translogDeletionPolicyOverridingPlugin = null;
        for (EnginePlugin enginePlugin : enginePlugins) {
            // get overriding codec service from EnginePlugin
            if (codecService.isPresent() == false) {
                codecService = enginePlugin.getCustomCodecService(idxSettings);
                codecServiceOverridingPlugin = enginePlugin.getClass().getName();
-            } else {
+            } else if (enginePlugin.getCustomCodecService(idxSettings).isPresent()) {


This seems to be a bug: the IllegalStateException is going to be thrown even if enginePlugin does not provide any codec service (returns Optional.empty).

Seems bad. Has this case just never been hit because in practice there is only ever 1 engine plugin or because the last plugin in the list is the one to override the codec service?

Hard to say, but I would guess that only handful of engine plugins do provide codec service (and probably there is usually only 1 as you mentioned).

Right, and I believe this change was introduced in 1.2 as well

reta · 2022-02-18T20:50:19Z

server/src/main/java/org/opensearch/index/engine/EngineConfigFactory.java

@@ -76,16 +88,42 @@ public EngineConfigFactory(PluginsService pluginsService, IndexSettings idxSetti
            if (translogDeletionPolicyFactory.isPresent() == false) {
                translogDeletionPolicyFactory = enginePlugin.getCustomTranslogDeletionPolicyFactory();
                translogDeletionPolicyOverridingPlugin = enginePlugin.getClass().getName();
-            } else {
+            } else if (enginePlugin.getCustomTranslogDeletionPolicyFactory().isPresent()) {


server/src/main/java/org/opensearch/plugins/EnginePlugin.java

opensearch-ci-bot · 2022-02-18T21:27:09Z

✅ Gradle Check success f063a50320163b6b3415415ccba66f4e27e7bcee
Log 2608

Reports 2608

andrross · 2022-02-19T00:47:41Z

@jmazanec15 Can you take a look at this and confirm it will solve the problem you described in #1805?

jmazanec15 · 2022-03-03T19:27:21Z

@andrross Apologies, just saw this tag. Will take a look this week.

jmazanec15 · 2022-03-03T22:51:01Z

server/src/main/java/org/opensearch/index/engine/EngineConfigFactory.java

        Optional<TranslogDeletionPolicyFactory> translogDeletionPolicyFactory = Optional.empty();
        String translogDeletionPolicyOverridingPlugin = null;
        for (EnginePlugin enginePlugin : enginePlugins) {
            // get overriding codec service from EnginePlugin
            if (codecService.isPresent() == false) {
                codecService = enginePlugin.getCustomCodecService(idxSettings);
                codecServiceOverridingPlugin = enginePlugin.getClass().getName();
-            } else {
+            } else if (enginePlugin.getCustomCodecService(idxSettings).isPresent()) {


Right, and I believe this change was introduced in 1.2 as well

jmazanec15 · 2022-03-03T22:52:16Z

server/src/main/java/org/opensearch/indices/IndicesService.java

@@ -742,7 +742,8 @@ private synchronized IndexService createIndexService(
    }

    private EngineConfigFactory getEngineConfigFactory(final IndexSettings idxSettings) {
-        return new EngineConfigFactory(this.pluginsService, idxSettings);
+        final IndexService indexService = indices.get(idxSettings.getUUID());


When would IndexService be null? If it is should it fail?

This is good question - I cannot say exactly, but since indices is just map, the get may return null, better be ready for it.

Im wondering if this will always be null?

getEngineConfigFactory gets called in createIndexService , which gets called in createIndex, but indices is not updated until after.

The mapper service will get created in IndexService's constructor.

The CodecService that is used when EnginePlugins are not used is actually created in IndexShard's constructor. Eventually, there is this somewhat hack in EngineConfigFactory.newEngineConfig where if there is a CodecService set in the EngineConfigFactory, it is used, otherwise the passed in one is used.

But I believe that the mapper service will not have been created at this point.

I think my opinion is that we should only create a codec service in one place and it should be here: https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/index/shard/IndexShard.java#L324. We should pass in the CodecService factory there and get rid of the codec service in EngineConfigFactory.

Very good point, looking into it now, thank you!

Could we do something like this: jmazanec15@165f50a? This way, we can exclusively use CodecServiceFactory to build codec and not change any function interfaces?

Alternatively, I thought about building CodecService directly in newEngineConfig. In this case, we would switch from passing in CodecService directly to passing in CodecServiceConfig. But, I thought this might cause us to create more CodecServices than we need to.

@jmazanec15 hm ... to me, the logger service does not belong here, may be if you share use case, we could think about it?

@reta Logger is one of the constructor arguments to CodecService. I think the CodecServiceConfig will need to have it to build CodecService - similar to MapperService. That way, we can use CodecServiceFactory to build default CodecService.

@jmazanec15 I should have looked it :) Thank you, will do the change shortly

Done, thank @jmazanec15 !

server/src/main/java/org/opensearch/plugins/EnginePlugin.java

opensearch-ci-bot · 2022-03-04T14:06:11Z

❌ Gradle Check failure c2827c35b89705456515e1c9626f78315fe7359b
Log 3026

Reports 3026

…ice constructor Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

opensearch-ci-bot · 2022-03-04T14:38:28Z

❌ Gradle Check failure 30b9b5085d208f7b0413268a5a9d11360870ccf5
Log 3027

Reports 3027

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

opensearch-ci-bot · 2022-03-04T15:57:20Z

✅ Gradle Check success 17c0abb
Log 3030

Reports 3030

andrross

LGTM! I'll just wait for sign off from @jmazanec15 before merging.

jmazanec15 · 2022-03-04T18:35:53Z

server/src/main/java/org/opensearch/index/codec/CodecServiceConfig.java

+ * The configuration parameters necessary for the {@link CodecService} instance construction.
+ */
+public final class CodecServiceConfig {
+    private final IndexSettings indexSettings;


If we are passing index settings to the getCustomCodecServiceFactory, do we need to include them in CodecServiceConfig?

Im just wondering if we should steer plugin developers to make decisions based on index settings in getCustomCodecServiceFactory instead of in createCodecService. Would this limit functionality? Does it really matter where decisions on index settings are being made?

I think I am leaning towards limiting it to getCustomCodecServiceFactory. What are your thoughts @nknize @reta?

We could, I believe the presence of the index settings could be beneficial, otherwise would leave that up to implementors to pass the index settings from EnginePlugin to CodecService instance if needed

jmazanec15 · 2022-03-04T18:51:18Z

server/src/main/java/org/opensearch/indices/IndicesService.java

@@ -742,7 +742,8 @@ private synchronized IndexService createIndexService(
    }

    private EngineConfigFactory getEngineConfigFactory(final IndexSettings idxSettings) {
-        return new EngineConfigFactory(this.pluginsService, idxSettings);
+        final IndexService indexService = indices.get(idxSettings.getUUID());


Im wondering if this will always be null?

getEngineConfigFactory gets called in createIndexService , which gets called in createIndex, but indices is not updated until after.

The mapper service will get created in IndexService's constructor.

The CodecService that is used when EnginePlugins are not used is actually created in IndexShard's constructor. Eventually, there is this somewhat hack in EngineConfigFactory.newEngineConfig where if there is a CodecService set in the EngineConfigFactory, it is used, otherwise the passed in one is used.

But I believe that the mapper service will not have been created at this point.

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

opensearch-ci-bot · 2022-03-04T21:54:47Z

❌ Gradle Check failure 8fb11a5a874f70cc033e2337ad0e1590a41b9af6
Log 3056

Reports 3056

opensearch-ci-bot · 2022-03-04T22:22:12Z

✅ Gradle Check success 5c136be
Log 3057

Reports 3057

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

opensearch-ci-bot · 2022-03-07T15:57:02Z

✅ Gradle Check success eeeab29
Log 3111

Reports 3111

dblock · 2022-03-07T16:49:44Z

@jmazanec15 you're good with this?

jmazanec15 · 2022-03-07T20:50:36Z

server/src/main/java/org/opensearch/index/engine/EngineConfigFactory.java

        this.translogDeletionPolicyFactory = translogDeletionPolicyFactory.orElse((idxs, rtls) -> null);
    }

-    /** Instantiates a new EngineConfig from the provided custom overrides */


Why cant we just delete this method and go with 179? This method looks to only be used IndexShard. Is it possible for plugin developers to access EngineConfigFactory.java?

This is a public method which could be used (by plugins fe) so we should not break them

jmazanec15 · 2022-03-08T01:43:42Z

server/src/main/java/org/opensearch/index/engine/EngineConfigFactory.java

@@ -124,7 +212,9 @@ public EngineConfig newEngineConfig(
            mergePolicy,
            analyzer,
            similarity,
-            this.codecService != null ? this.codecService : codecService,
+            this.codecServiceFactory != null
+                ? this.codecServiceFactory.createCodecService(new CodecServiceConfig(indexSettings, mapperService, logger))


I think we should only build CodecService in one place and always use the factory. That way, we can avoid ternary statements like the above. For default CodecServiceFactory, we can set it like I did here.

Im somewhat conflicted on where to build CodecService. I think there are 2 options:

1. Call this.codecServiceFactory.createCodecService here and dont take in CodecService as arg to newEngineConfig

The pro's to this approach are that we can hide details about CodecService from all other parts of code - in IndexShard we dont need to do anything with codec service. Its all handled by EngineConfigFactory.

The cons are that a new CodecService would get created every time this method gets called. I'm not really sure if there will be side effects to this. In IndexShard, this gets called in resetEngineToGlobalCheckpoint and innerOpenEngineAndTranslog. One potential issue could arise if a user changes IndexSettings between these 2 operations and that impacts how the CodecServiceFactory builds the CodecService (maybe this is a feature not a bug?).

2. Call it in IndexShard's constructor by exposing getter on EngineConfigFactory's codecServiceFactory

Alternatively, we could create the codec service in IndexShards constructor. This is how it is being done now for the default. To do this, we would need to expose a getter method for EngineConfigFactory to get its codecServiceFactory and then call this.codecServiceFactory.createCodecService(). Then, we wouldnt change anything in EngineConfigFactory.newEngineConfig(). The getter, however, feels a little bit out of place here. I took this approach here.

I think I am leaning towards approach 2 as opposed to 1, to keep consistent with how its being done now.

@jmazanec15 I think I got your idea, the challenge here is to preserve the semantics of the CodecService instantiation, I just pushed the solution which I believe is the best effort:

new method EngineConfigFactory::newCodecServiceOrDefault which basically either create a new CodecService instance from factory or uses the existing one

IndexShard::newEngineConfig uses EngineConfigFactory::newCodecServiceOrDefault to pass the CodecService instance to EngineConfigFactory::newEngineConfig

EngineConfigFactory::newEngineConfig has the fallback in case passed CodecService is null but there is CodecServiceFactory configured - the old way to create CodecService without MapperService and Logger

The APIs stay unchanged, thank you

That makes sense thanks!

opensearch-ci-bot · 2022-03-08T15:28:11Z

❌ Gradle Check failure 164b6d91a3ce13e1023b1f1253a72af290c30930
Log 3165

Reports 3165

…CodecService Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

opensearch-ci-bot · 2022-03-08T16:14:23Z

✅ Gradle Check success 5379899
Log 3166

Reports 3166

dblock · 2022-03-08T18:13:40Z

If @jmazanec15 is good with this change we can merge, CR please?

jmazanec15 · 2022-03-08T20:18:56Z

Looks good @dblock

…ice constructor (#2177) * MapperService has to be passed in as null for EnginePlugins CodecService constructor Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Addressing code review comments Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Delayed CodecService instantiation up to the shard initialization Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Added logger (associated with shard) to CodecServiceConfig Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Refactored the EngineConfigFactory / IndexShard instantiation of the CodecService Signed-off-by: Andriy Redko <andriy.redko@aiven.io> (cherry picked from commit 9c679cb)

…ice constructor (#2177) (#2413) * MapperService has to be passed in as null for EnginePlugins CodecService constructor Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Addressing code review comments Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Delayed CodecService instantiation up to the shard initialization Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Added logger (associated with shard) to CodecServiceConfig Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Refactored the EngineConfigFactory / IndexShard instantiation of the CodecService Signed-off-by: Andriy Redko <andriy.redko@aiven.io> (cherry picked from commit 9c679cb) Co-authored-by: Andriy Redko <andriy.redko@aiven.io>

This includes the following: commit 9cfa395 Author: Kartik <gkart@amazon.com> Date: Thu Mar 10 10:12:17 2022 -0800 Remove the IndexCommitRef class (opensearch-project#2421) This inner class is no longer required because its functionality has been moved to the generic GatedCloseable class. Signed-off-by: Kartik Ganesh <gkart@amazon.com> commit c8d8009 Author: Andriy Redko <andriy.redko@aiven.io> Date: Thu Mar 10 11:46:08 2022 -0500 Fixing bwcVersions and bwc builds (opensearch-project#2430) - adding 1.4.0 into main bwcVersions Signed-off-by: Andriy Redko <andriy.redko@aiven.io> commit fb9e150 Author: Kartik <85275476+kartg@users.noreply.github.com> Date: Wed Mar 9 12:21:09 2022 -0800 Refactoring gated and ref-counted interfaces and their implementations (opensearch-project#2396) * Reducing duplication in plugins around ref-counted releasable classes Both AmazonEc2Reference and AmazonS3Reference duplicate the same logic - a subclass of AbstractRefCounted that also implements Releasable. This change centralizes this paradigm into a AbstractRefCountedReleasable class and supports both clients via generics. It also updates all fetching implementations to use the get() method instead of client() Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Introduce Reference classes for the Closeable and AutoCloseable interfaces These classes allow you to wrap a reference instance with an onClose runnable that is executed when close() is invoked. Two separate classes are needed because the close() signatures for the two interfaces are different. This change takes the first step to have implementing classes extend from these generic superclasses, before attempting to remove the subclasses entirely. The get() method is also replaced throughout the code base. Note that there is also a separate Releasable interface that has a similar access pattern, but is implemented separately. This is used in AbstractRefCountedReleasable introduced in a prior commit Signed-off-by: Kartik Ganesh <gkart@amazon.com> * More improvements and refactoring * Functionality around one-way gating is now moved to a dedicated class - OneWayGate. This replaces duplicate functionality throughout the code. * The two *Reference classes have been renamed to Gated* since that better represents their functionality * The AbstractRefCountedReleasable has been improved to no longer be abstract by accepting the shutdown hook. This removes the need for the inner class in ReleasableBytesReference, and further simplifies the plugin subclasses (these could probably be removed entirely). * Finally, unit tests have been added for some classes Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Added tests for GatedCloseable Also updated the license information in GatedAutoCloseableTests Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Fixing license information in new files Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Added unit tests for RefCountedReleasable Signed-off-by: Kartik Ganesh <gkart@amazon.com> commit 5a9a114 Author: Nick Knize <nknize@apache.org> Date: Wed Mar 9 12:50:05 2022 -0600 [Remove] TrimUnsafeCommit logic for legacy 6.x indexes (opensearch-project#2225) * [Remove] TrimUnsafeCommit logic for legacy 6.x indexes Multiple txlog commits was introduced in legacy 7.x. Legacy 6.x indexes could therefore not have a safe commit. Since OpenSearch 2.0 is no longer compatible with legacy 6.x indexes, the logic to trim these unsafe commits is safely removed. Signed-off-by: Nicholas Walter Knize <nknize@apache.org> * fix assertion typo Signed-off-by: Nicholas Walter Knize <nknize@apache.org> * rebase and incorporate pr feedback Signed-off-by: Nicholas Walter Knize <nknize@apache.org> commit 9c679cb Author: Andriy Redko <andriy.redko@aiven.io> Date: Tue Mar 8 18:42:32 2022 -0500 MapperService has to be passed in as null for EnginePlugins CodecService constructor (opensearch-project#2177) * MapperService has to be passed in as null for EnginePlugins CodecService constructor Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Addressing code review comments Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Delayed CodecService instantiation up to the shard initialization Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Added logger (associated with shard) to CodecServiceConfig Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Refactored the EngineConfigFactory / IndexShard instantiation of the CodecService Signed-off-by: Andriy Redko <andriy.redko@aiven.io> commit a6a47e7 Author: Suraj Singh <79435743+dreamer-89@users.noreply.github.com> Date: Tue Mar 8 14:43:04 2022 -0800 Remove inclue_type_name parameter from rest api spec (opensearch-project#2410) Signed-off-by: Suraj Singh <surajrider@gmail.com> commit 044f536 Author: Daniel Doubrovkine (dB.) <dblock@dblock.org> Date: Tue Mar 8 14:48:51 2022 -0500 Set target and source compatibility to 11, required by Lucene 9. (opensearch-project#2407) * Set target and source compatibility to 11, required by Lucene 9. Signed-off-by: dblock <dblock@dblock.org> * Uncomment commented code in opensearch-project#2321 for killing child processes that uses JDK9+ ProcessInfo. Signed-off-by: dblock <dblock@dblock.org> * Set distribution checker target JDK compatibility to 11. Signed-off-by: dblock <dblock@dblock.org> * Supress processing warnings. Signed-off-by: dblock <dblock@dblock.org> commit c3712a5 Author: Nick Knize <nknize@apache.org> Date: Tue Mar 8 11:30:27 2022 -0600 [Remove] include_type_name from HLRC (opensearch-project#2397) Removes include_type_name from the high level reset client along with relevant deprecated methods in IndicesClient. All tests are updated to remove the parameter from the rest requests along with various toXContent methods that are no longer required. Signed-off-by: Nicholas Walter Knize <nknize@apache.org> commit 63c75d1 Author: Tianli Feng <ftianli@amazon.com> Date: Tue Mar 8 08:35:36 2022 -0800 Deprecate setting 'reindex.remote.whitelist' and introduce the alternative setting 'reindex.remote.allowlist' (opensearch-project#2221) * Add setting reindex.remote.allowlist, and deprecate setting reindex.remote.whitelist Signed-off-by: Tianli Feng <ftianli@amazon.com> * Add unit test for renaming the setting reindex.remote.allowlist Signed-off-by: Tianli Feng <ftianli@amazon.com> * Remove system.out.println() Signed-off-by: Tianli Feng <ftianli@amazon.com> * Adjust format by spotlessApply task Signed-off-by: Tianli Feng <ftianli@amazon.com> * Replace REMOTE_CLUSTER_WHITELIST with REMOTE_CLUSTER_ALLOWLIST Signed-off-by: Tianli Feng <ftianli@amazon.com> * Add a unit test to test final setting value when both settings have got a value Signed-off-by: Tianli Feng <ftianli@amazon.com> * Rename the unit test class name Signed-off-by: Tianli Feng <ftianli@amazon.com> * Remove the Access modifiers public from the constant REMOTE_CLUSTER_WHITELIST Signed-off-by: Tianli Feng <ftianli@amazon.com> * Initialize ReindexPlugin without using the @before method Signed-off-by: Tianli Feng <ftianli@amazon.com> * Rename 'unwhitelisted' to 'unallowlisted' in a yml file used for REST api testing. Signed-off-by: Tianli Feng <ftianli@amazon.com> commit 65debde Author: Andriy Redko <andriy.redko@aiven.io> Date: Tue Mar 8 11:30:48 2022 -0500 Update the BWC versions (post 1.x backport) (opensearch-project#2390) Signed-off-by: Andriy Redko <andriy.redko@aiven.io> commit 919d180 Author: Suraj Singh <79435743+dreamer-89@users.noreply.github.com> Date: Mon Mar 7 12:43:05 2022 -0800 Remove type end-points from count action (opensearch-project#2379) Signed-off-by: Suraj Singh <surajrider@gmail.com> commit 1f0361a Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon Mar 7 10:28:17 2022 -0800 Bump asm-commons from 5.0.4 to 9.2 in /modules/lang-expression (opensearch-project#2385) * Bump asm-commons from 5.0.4 to 9.2 in /modules/lang-expression Bumps asm-commons from 5.0.4 to 9.2. --- updated-dependencies: - dependency-name: org.ow2.asm:asm-commons dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * Updating SHAs Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> commit 09e16e3 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon Mar 7 13:08:07 2022 -0500 Bump guava from 30.1.1-jre to 31.1-jre in /plugins/repository-azure (opensearch-project#2382) * Bump guava from 30.1.1-jre to 31.1-jre in /plugins/repository-azure Bumps [guava](https://github.com/google/guava) from 30.1.1-jre to 31.1-jre. - [Release notes](https://github.com/google/guava/releases) - [Commits](https://github.com/google/guava/commits) --- updated-dependencies: - dependency-name: com.google.guava:guava dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * Updating SHAs Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> commit e1fd4b7 Author: Subhobrata Dey <sbcd90@gmail.com> Date: Mon Mar 7 08:51:49 2022 -0800 Add valuesField in PercentilesAggregationBuilder streamInput constructor (opensearch-project#2308) Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> commit 4395ed5 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon Mar 7 11:48:38 2022 -0500 Bump guava in /distribution/tools/upgrade-cli (opensearch-project#2383) Bumps [guava](https://github.com/google/guava) from 31.0.1-jre to 31.1-jre. - [Release notes](https://github.com/google/guava/releases) - [Commits](https://github.com/google/guava/commits) --- updated-dependencies: - dependency-name: com.google.guava:guava dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit 72c5d81 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon Mar 7 11:48:18 2022 -0500 Bump guava in /distribution/tools/keystore-cli (opensearch-project#2384) Bumps [guava](https://github.com/google/guava) from 31.0.1-jre to 31.1-jre. - [Release notes](https://github.com/google/guava/releases) - [Commits](https://github.com/google/guava/commits) --- updated-dependencies: - dependency-name: com.google.guava:guava dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit 75e837d Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon Mar 7 11:48:00 2022 -0500 Bump guava from 31.0.1-jre to 31.1-jre in /distribution/tools/plugin-cli (opensearch-project#2387) Bumps [guava](https://github.com/google/guava) from 31.0.1-jre to 31.1-jre. - [Release notes](https://github.com/google/guava/releases) - [Commits](https://github.com/google/guava/commits) --- updated-dependencies: - dependency-name: com.google.guava:guava dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit 3e9031f Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon Mar 7 11:47:37 2022 -0500 Bump gradle-extra-configurations-plugin from 3.0.3 to 7.0.0 in /buildSrc (opensearch-project#2386) Bumps [gradle-extra-configurations-plugin](https://github.com/nebula-plugins/gradle-extra-configurations-plugin) from 3.0.3 to 7.0.0. - [Release notes](https://github.com/nebula-plugins/gradle-extra-configurations-plugin/releases) - [Changelog](https://github.com/nebula-plugins/gradle-extra-configurations-plugin/blob/main/CHANGELOG.md) - [Commits](nebula-plugins/gradle-extra-configurations-plugin@v3.0.3...v7.0.0) --- updated-dependencies: - dependency-name: com.netflix.nebula:gradle-extra-configurations-plugin dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit 9224537 Author: Owais Kazi <owaiskazi19@gmail.com> Date: Fri Mar 4 16:58:13 2022 -0800 Updated the url for docker distribution (opensearch-project#2325) Signed-off-by: Owais Kazi <owaiskazi19@gmail.com> commit d5e58a2 Author: Suraj Singh <79435743+dreamer-89@users.noreply.github.com> Date: Fri Mar 4 16:40:47 2022 -0800 [Remove] Type mappings from GeoShapeQueryBuilder (opensearch-project#2322) * Remove type end-points from GeoShapeBuilder Signed-off-by: Suraj Singh <surajrider@gmail.com> * Fix integration test failures Signed-off-by: Suraj Singh <surajrider@gmail.com> commit be64af2 Author: aponb <aponb@gmx.at> Date: Sat Mar 5 01:34:09 2022 +0100 Replace exclusionary words whitelist and blacklist in the places that won't impact backwards compatibility (opensearch-project#2178) * Replace the exclusionary word whitelist with allowlist, and blacklist with denylist, in code commet and internal variable/method/class/package name. Signed-off-by: Andreas <apre@gmx.at> commit ae52008 Author: Andriy Redko <andriy.redko@aiven.io> Date: Fri Mar 4 17:44:52 2022 -0500 Fixing the --release flag usage for javac (opensearch-project#2343) (opensearch-project#2352) * Fixing the --release flag usage for javac (opensearch-project#2343) * Fixing the --release flag usage for javac Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Fixing the --html5 flag usage for javadoc Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Fix java-version-checker source/target compatibility settings (opensearch-project#2354) Signed-off-by: Andriy Redko <andriy.redko@aiven.io> commit 0cc2c9b Author: Nick Knize <nknize@apache.org> Date: Fri Mar 4 13:30:43 2022 -0600 [Remove] types from PutMappingRequest (opensearch-project#2335) Remove type support from putMappingRequest, dependencies, and all tests. Signed-off-by: Nicholas Walter Knize <nknize@apache.org> commit 729bc43 Author: Nick Knize <nknize@apache.org> Date: Fri Mar 4 10:52:24 2022 -0600 [Test-Failure] Mute TranslogPolicyIT (opensearch-project#2342) This test is slated for removal as it only applies to ancient indexes (Legacy 6.x). Muting test so bwc tests are consistent and no longer angry. Signed-off-by: Nicholas Walter Knize <nknize@apache.org> commit 5f90227 Author: Andriy Redko <andriy.redko@aiven.io> Date: Fri Mar 4 11:12:27 2022 -0500 Add '_name' field support to score functions and provide it back in explanation response (opensearch-project#2244) * Add '_name' field support to score functions and provide it back in explanation response Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Address code review comments Signed-off-by: Andriy Redko <andriy.redko@aiven.io> commit ae14259 Author: Daniel Doubrovkine (dB.) <dblock@dblock.org> Date: Thu Mar 3 15:34:53 2022 -0500 Restore Java 8 compatibility for build tools. (opensearch-project#2300) (opensearch-project#2321) * Restore Java 8 compatibility for build tools. Signed-off-by: dblock <dblock@dblock.org> * Make source code compatible with Java 8. Signed-off-by: dblock <dblock@dblock.org> commit cb57b92 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu Mar 3 14:23:47 2022 -0500 Bump log4j-core in /buildSrc/src/testKit/thirdPartyAudit/sample_jars (opensearch-project#2281) Bumps log4j-core from 2.17.1 to 2.17.2. --- updated-dependencies: - dependency-name: org.apache.logging.log4j:log4j-core dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit cdb42ad Author: Vacha Shah <vachshah@amazon.com> Date: Thu Mar 3 10:39:27 2022 -0800 Remove Github DCO action since DCO runs via Github App now (opensearch-project#2317) Signed-off-by: Vacha Shah <vachshah@amazon.com> commit f13b951 Author: Andrey Pleskach <ples@aiven.io> Date: Wed Mar 2 23:36:09 2022 +0100 Add support of SOCKS proxies for S3 repository (opensearch-project#2160) Signed-off-by: Andrey Pleskach <ples@aiven.io> commit 3d5aff4 Author: Suraj Singh <79435743+dreamer-89@users.noreply.github.com> Date: Wed Mar 2 13:21:00 2022 -0800 Remove type end-points from search and related APIs (opensearch-project#2263) Signed-off-by: Suraj Singh <surajrider@gmail.com> commit 897f4e7 Author: Nick Knize <nknize@apache.org> Date: Wed Mar 2 13:44:04 2022 -0600 [Remove] deprecated getMapping API from IndicesClient (opensearch-project#2262) Removes the deprecated types based get, getMapping, getAsync, and getMappingAsync methods from IndicesClient. It also removes extra nesting of mappings belong the deprecated type named object and removes the types based methods from the affected request classes. Signed-off-by: Nicholas Walter Knize <nknize@apache.org> commit 4b89410 Author: Breno Faria <30877433+br3no@users.noreply.github.com> Date: Wed Mar 2 19:52:38 2022 +0100 Reintroduce negative epoch_millis opensearch-project#1991 (opensearch-project#2232) * Reintroduce negative epoch_millis opensearch-project#1991 Fixes a regression introduced with Elasticsearch 7 regarding the date field type that removed support for negative timestamps with sub-second granularity. Thanks to Ryan Kophs (https://github.com/rkophs) for allowing me to use his previous work. Signed-off-by: Breno Faria <breno.faria@intrafind.de> * applying spotless fix Signed-off-by: Breno Faria <breno.faria@intrafind.de> * more conservative implementation of isSupportedBy Signed-off-by: Breno Faria <breno.faria@intrafind.de> * adding braces to control flow statement Signed-off-by: Breno Faria <breno.faria@intrafind.de> * spotless fix... Signed-off-by: Breno Faria <breno.faria@intrafind.de> Co-authored-by: Breno Faria <breno.faria@intrafind.de> commit 9e225dc Author: Peng Huo <penghuo@gmail.com> Date: Wed Mar 2 10:34:02 2022 -0800 Fix flaky test case - string profiler via global ordinals (opensearch-project#2226) forcemerge to one segment before executing aggregation query. Signed-off-by: Peng Huo <penghuo@gmail.com> commit c8a7606 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon Feb 28 12:18:28 2022 -0800 Bump commons-math3 from 3.2 to 3.6.1 in /benchmarks (opensearch-project#2282) Bumps commons-math3 from 3.2 to 3.6.1. --- updated-dependencies: - dependency-name: org.apache.commons:commons-math3 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit 1b8181c Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon Feb 28 12:17:16 2022 -0800 Bump gson from 2.8.9 to 2.9.0 in /plugins/repository-hdfs (opensearch-project#2279) * Bump gson from 2.8.9 to 2.9.0 in /plugins/repository-hdfs Bumps [gson](https://github.com/google/gson) from 2.8.9 to 2.9.0. - [Release notes](https://github.com/google/gson/releases) - [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md) - [Commits](google/gson@gson-parent-2.8.9...gson-parent-2.9.0) --- updated-dependencies: - dependency-name: com.google.code.gson:gson dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Updating SHAs Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> commit 0df9845 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon Feb 28 12:16:56 2022 -0800 Bump morfologik-fsa from 2.1.1 to 2.1.8 in /plugins/analysis-ukrainian (opensearch-project#2278) * Bump morfologik-fsa from 2.1.1 to 2.1.8 in /plugins/analysis-ukrainian Bumps [morfologik-fsa](https://github.com/morfologik/morfologik-stemming) from 2.1.1 to 2.1.8. - [Release notes](https://github.com/morfologik/morfologik-stemming/releases) - [Changelog](https://github.com/morfologik/morfologik-stemming/blob/master/CHANGES.txt) - [Commits](morfologik/morfologik-stemming@2.1.1...2.1.8) --- updated-dependencies: - dependency-name: org.carrot2:morfologik-fsa dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * Updating SHAs Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> commit 9780fc6 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon Feb 28 12:16:34 2022 -0800 Bump bc-fips from 1.0.2.1 to 1.0.2.3 in /distribution/tools/plugin-cli (opensearch-project#2276) * Bump bc-fips from 1.0.2.1 to 1.0.2.3 in /distribution/tools/plugin-cli Bumps bc-fips from 1.0.2.1 to 1.0.2.3. --- updated-dependencies: - dependency-name: org.bouncycastle:bc-fips dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * Updating SHAs Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> commit 4ef30f4 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon Feb 28 10:41:46 2022 -0800 Bump azure-storage-common from 12.14.0 to 12.14.3 in /plugins/repository-azure (opensearch-project#2274) * Bump azure-storage-common in /plugins/repository-azure Bumps [azure-storage-common](https://github.com/Azure/azure-sdk-for-java) from 12.14.0 to 12.14.3. - [Release notes](https://github.com/Azure/azure-sdk-for-java/releases) - [Commits](Azure/azure-sdk-for-java@azure-storage-blob_12.14.0...azure-storage-blob_12.14.3) --- updated-dependencies: - dependency-name: com.azure:azure-storage-common dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * Updating SHAs Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> commit 5d0b015 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon Feb 28 10:41:33 2022 -0800 Bump jimfs from 1.1 to 1.2 in /distribution/tools/keystore-cli (opensearch-project#2272) Bumps [jimfs](https://github.com/google/jimfs) from 1.1 to 1.2. - [Release notes](https://github.com/google/jimfs/releases) - [Commits](google/jimfs@v1.1...v1.2) --- updated-dependencies: - dependency-name: com.google.jimfs:jimfs dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit f6264a9 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon Feb 28 10:41:18 2022 -0800 Bump spock-core from 2.0-groovy-3.0 to 2.1-groovy-3.0 in /buildSrc (opensearch-project#2270) Bumps spock-core from 2.0-groovy-3.0 to 2.1-groovy-3.0. --- updated-dependencies: - dependency-name: org.spockframework:spock-core dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit 21f11ec Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon Feb 28 10:40:51 2022 -0800 Bump asm-tree from 5.0.4 to 9.2 in /modules/lang-expression (opensearch-project#2269) * Bump asm-tree from 5.0.4 to 9.2 in /modules/lang-expression Bumps asm-tree from 5.0.4 to 9.2. --- updated-dependencies: - dependency-name: org.ow2.asm:asm-tree dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * Updating SHAs Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> commit 223d0b2 Author: Suraj Singh <79435743+dreamer-89@users.noreply.github.com> Date: Fri Feb 25 13:00:25 2022 -0800 Remove type end-points from no-op bulk and search action (opensearch-project#2261) Signed-off-by: Suraj Singh <surajrider@gmail.com> commit 0bd7850 Author: Nick Knize <nknize@apache.org> Date: Fri Feb 25 13:35:48 2022 -0600 [Remove] remaining type usage in Client and AbstractClient (opensearch-project#2258) Removes type parameter from remaining prepareIndex in Client and AbstractClient. Signed-off-by: Nicholas Walter Knize <nknize@apache.org> commit 3a4c2f6 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Fri Feb 25 10:40:11 2022 -0800 Bump jimfs from 1.1 to 1.2 in /qa/evil-tests (opensearch-project#2130) * Bump jimfs from 1.1 to 1.2 in /qa/evil-tests Bumps [jimfs](https://github.com/google/jimfs) from 1.1 to 1.2. - [Release notes](https://github.com/google/jimfs/releases) - [Commits](google/jimfs@v1.1...v1.2) --- updated-dependencies: - dependency-name: com.google.jimfs:jimfs dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Fixing failing precommit and check Signed-off-by: Vacha Shah <vachshah@amazon.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Vacha Shah <vachshah@amazon.com> commit 5b0da85 Author: Suraj Singh <79435743+dreamer-89@users.noreply.github.com> Date: Fri Feb 25 10:18:41 2022 -0800 Remove type from validate query API (opensearch-project#2255) * Remove type mapping from RestValidateAction Signed-off-by: Suraj Singh <surajrider@gmail.com> * Spotless check apply Signed-off-by: Suraj Singh <surajrider@gmail.com> * Include suggested review comment Signed-off-by: Suraj Singh <surajrider@gmail.com> commit 494c7bc Author: Rishikesh Pasham <62345295+Rishikesh1159@users.noreply.github.com> Date: Fri Feb 25 03:39:43 2022 +0000 Revert "Override Default Distribution Download Url with Custom Distribution Url When User Passes a Url" (opensearch-project#2256) * Override default Distribution Download URL with custom Distribution URL Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * Accidently made commit to main branch, this revives it.Override default Distribution Download URL with custom Distribution URL Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * Revert Override Default Distribution Download Url with Custom Distribution Url When User Passes a Url Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> commit 8b48207 Author: Nick Knize <nknize@apache.org> Date: Thu Feb 24 21:20:03 2022 -0600 [Remove] Type from Client.prepare(Index,Delete,Update) (opensearch-project#2253) Removes the type parameter from Client.prepare(Index,Delete,Update) and everywhere it's used throughout the codebase except for prepareIndex(index, type, id) which is removed in a follow up. Signed-off-by: Nicholas Walter Knize <nknize@apache.org> Merge branch 'main' into feature/segment-replication Signed-off-by: Kartik Ganesh <gkart@amazon.com>

reta requested a review from a team as a code owner February 18, 2022 20:48

reta commented Feb 18, 2022

View reviewed changes

server/src/main/java/org/opensearch/plugins/EnginePlugin.java Outdated Show resolved Hide resolved

jmazanec15 reviewed Mar 3, 2022

View reviewed changes

MapperService has to be passed in as null for EnginePlugins CodecServ…

45665db

…ice constructor Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

reta force-pushed the issue-1805 branch from c2827c3 to 30b9b50 Compare March 4, 2022 14:15

Addressing code review comments

17c0abb

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

reta force-pushed the issue-1805 branch from 30b9b50 to 17c0abb Compare March 4, 2022 15:11

andrross approved these changes Mar 4, 2022

View reviewed changes

jmazanec15 reviewed Mar 4, 2022

View reviewed changes

Delayed CodecService instantiation up to the shard initialization

5c136be

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

reta force-pushed the issue-1805 branch from 8fb11a5 to 5c136be Compare March 4, 2022 21:40

Added logger (associated with shard) to CodecServiceConfig

eeeab29

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

jmazanec15 reviewed Mar 8, 2022

View reviewed changes

Refactored the EngineConfigFactory / IndexShard instantiation of the …

5379899

…CodecService Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

reta force-pushed the issue-1805 branch from 164b6d9 to 5379899 Compare March 8, 2022 15:35

jmazanec15 approved these changes Mar 8, 2022

View reviewed changes

dblock merged commit 9c679cb into opensearch-project:main Mar 8, 2022

dblock added the backport 1.x label Mar 8, 2022

opensearch-trigger-bot bot mentioned this pull request Mar 8, 2022

[Backport 1.x] MapperService has to be passed in as null for EnginePlugins CodecService constructor #2413

Merged

MapperService has to be passed in as null for EnginePlugins CodecService constructor #2177

MapperService has to be passed in as null for EnginePlugins CodecService constructor #2177

Conversation

reta commented Feb 18, 2022

Description

Issues Resolved

Check List

opensearch-ci-bot commented Feb 18, 2022

reta Feb 18, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

reta Feb 18, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

opensearch-ci-bot commented Feb 18, 2022

andrross commented Feb 19, 2022

jmazanec15 commented Mar 3, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

reta Mar 4, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

opensearch-ci-bot commented Mar 4, 2022

opensearch-ci-bot commented Mar 4, 2022

opensearch-ci-bot commented Mar 4, 2022

andrross left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

reta Mar 4, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

opensearch-ci-bot commented Mar 4, 2022

opensearch-ci-bot commented Mar 4, 2022

opensearch-ci-bot commented Mar 7, 2022

dblock commented Mar 7, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

1. Call this.codecServiceFactory.createCodecService here and dont take in CodecService as arg to newEngineConfig

2. Call it in IndexShard's constructor by exposing getter on EngineConfigFactory's codecServiceFactory

Choose a reason for hiding this comment

Choose a reason for hiding this comment

opensearch-ci-bot commented Mar 8, 2022

opensearch-ci-bot commented Mar 8, 2022

dblock commented Mar 8, 2022 • edited Loading

jmazanec15 commented Mar 8, 2022

reta Feb 18, 2022 •

edited

Loading

reta Feb 18, 2022 •

edited

Loading

reta Mar 4, 2022 •

edited

Loading

reta Mar 4, 2022 •

edited

Loading

1. Call `this.codecServiceFactory.createCodecService` here and dont take in CodecService as arg to newEngineConfig

dblock commented Mar 8, 2022 •

edited

Loading