Merge remote-tracking branch 'upstream/2.x' into weighted-routing/bac…

…kport2.x

CHANGELOG.md

@@ -24,7 +24,10 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 - Added missing no-jdk distributions ([#4722](https://github.com/opensearch-project/OpenSearch/pull/4722))
 - Copy `build.sh` over from opensearch-build ([#4887](https://github.com/opensearch-project/OpenSearch/pull/4887))
 - Update GeoGrid base class access modifier to support extensibility ([#4921](https://github.com/opensearch-project/OpenSearch/pull/4921))
+- Recommission API changes for service layer ([#4320](https://github.com/opensearch-project/OpenSearch/pull/4320))
+- Recommissioning of zone. REST layer support. ([#4624](https://github.com/opensearch-project/OpenSearch/pull/4604))
 - Build no-jdk distributions as part of release build ([#4902](https://github.com/opensearch-project/OpenSearch/pull/4902))
+- Use getParameterCount instead of getParameterTypes ([#4821](https://github.com/opensearch-project/OpenSearch/pull/4821))
 
 ### Dependencies
 - Bumps `com.diffplug.spotless` from 6.9.1 to 6.10.0
@@ -40,6 +43,8 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 - Update Apache Lucene to 9.4.1 ([#4922](https://github.com/opensearch-project/OpenSearch/pull/4922))
 - Bump `woodstox-core` to 6.4.0 ([#4951](https://github.com/opensearch-project/OpenSearch/pull/4951))
 - Upgrade jetty-http, kotlin-stdlib and snakeyaml ([#4982](https://github.com/opensearch-project/OpenSearch/pull/4982))
+- OpenJDK Update (October 2022 Patch releases) ([#4997](https://github.com/opensearch-project/OpenSearch/pull/4997))
+- Upgrade zookeeper dependency in hdfs-fixture ([#5007](https://github.com/opensearch-project/OpenSearch/pull/5007))
 
 ### Changed
 - Dependency updates (httpcore, mockito, slf4j, httpasyncclient, commons-codec) ([#4308](https://github.com/opensearch-project/OpenSearch/pull/4308))
@@ -57,6 +62,9 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 - GET api for weighted shard routing([#4275](https://github.com/opensearch-project/OpenSearch/pull/4275/))
 - Delete api for weighted shard routing([#4400](https://github.com/opensearch-project/OpenSearch/pull/4400/))
 - Fix weighted routing metadata deserialization error on process restart ([#4691](https://github.com/opensearch-project/OpenSearch/pull/4691))
+- Add DecommissionService and helper to execute awareness attribute decommissioning ([#4084](https://github.com/opensearch-project/OpenSearch/pull/4084))
+- Add APIs (GET/PUT) to decommission awareness attribute ([#4261](https://github.com/opensearch-project/OpenSearch/pull/4261))
+- Controlling discovery for decommissioned nodes ([#4590](https://github.com/opensearch-project/OpenSearch/pull/4590))
 
 ### Deprecated
 ### Removed
@@ -82,9 +90,12 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 - [Bug]: Fixed invalid location of JDK dependency for arm64 architecture([#4613](https://github.com/opensearch-project/OpenSearch/pull/4613))
 - [Bug]: Alias filter lost after rollover ([#4499](https://github.com/opensearch-project/OpenSearch/pull/4499))
 - Fixing Gradle warnings associated with publishPluginZipPublicationToXxx tasks ([#4696](https://github.com/opensearch-project/OpenSearch/pull/4696))
+- [BUG]: Remove redundant field from GetDecommissionStateResponse ([#4751](https://github.com/opensearch-project/OpenSearch/pull/4751))
 - Fixed randomly failing test ([4774](https://github.com/opensearch-project/OpenSearch/pull/4774))
 - Fix recovery path for searchable snapshots ([4813](https://github.com/opensearch-project/OpenSearch/pull/4813))
 - Fix a bug on handling an invalid array value for point type field #4900([#4900](https://github.com/opensearch-project/OpenSearch/pull/4900))
+- Fix decommission status update to non leader nodes ([4800](https://github.com/opensearch-project/OpenSearch/pull/4800))
+- Fix bug in AwarenessAttributeDecommissionIT([4822](https://github.com/opensearch-project/OpenSearch/pull/4822))
 - Fix for failing checkExtraction, checkLicense and checkNotice tasks for windows gradle check ([#4941](https://github.com/opensearch-project/OpenSearch/pull/4941))
 ### Security
 - CVE-2022-25857 org.yaml:snakeyaml DOS vulnerability ([#4341](https://github.com/opensearch-project/OpenSearch/pull/4341))
@@ -106,12 +117,17 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 - Add dev guide for dealing with flaky tests ([4894](https://github.com/opensearch-project/OpenSearch/pull/4894))
 - Renamed flaky tests ([#4935](https://github.com/opensearch-project/OpenSearch/pull/4935))
 - Bump current version to 2.5.0 on 2.x branch ([#5002](https://github.com/opensearch-project/OpenSearch/pull/5002))
+- Remote shard balancer support for searchable snapshots ([#5024](https://github.com/opensearch-project/OpenSearch/pull/5024))
+- Implement Searchable Snapshot using OnDemandBlockIndexInput ([#4892](https://github.com/opensearch-project/OpenSearch/pull/4892))
 ### Dependencies
 - Update Jackson Databind to 2.13.4.2 (addressing CVE-2022-42003) ([#4781](https://github.com/opensearch-project/OpenSearch/pull/4781))
 - Install and configure Log4j JUL Adapter for Lucene 9.4 ([#4754](https://github.com/opensearch-project/OpenSearch/pull/4754))
 - Added feature to ignore indexes starting with dot during shard limit validation.([#4695](https://github.com/opensearch-project/OpenSearch/pull/4695))
+- Bump protobuf-java to 3.21.8 ([#5005](https://github.com/opensearch-project/OpenSearch/pull/5005))
+- Upgrade zookeeper dependency in hdfs-fixture ([#5047](https://github.com/opensearch-project/OpenSearch/pull/5047))
 ### Changed
 - Refactored BalancedAllocator.Balancer to LocalShardsBalancer ([#4818](https://github.com/opensearch-project/OpenSearch/pull/4818))
+- Skip SymbolicLinkPreservingTarIT when running on Windows ([#5023](https://github.com/opensearch-project/OpenSearch/pull/5023))
 ### Deprecated
 ### Removed
 - Remove RepositoryData.MIN_VERSION support for next major release ([4729](https://github.com/opensearch-project/OpenSearch/pull/4729))
@@ -131,6 +147,8 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 - Better plural stemmer than minimal_english ([#4738](https://github.com/opensearch-project/OpenSearch/pull/4738))
 - Fix AbstractStringFieldDataTestCase tests to account for TotalHits lower bound ([4867](https://github.com/opensearch-project/OpenSearch/pull/4867))
 - [Segment Replication] Fix bug of replica shard's translog not purging on index flush when segment replication is enabled ([4975](https://github.com/opensearch-project/OpenSearch/pull/4975))
+- Fix bug in SlicedInputStream with zero length ([#4863](https://github.com/opensearch-project/OpenSearch/pull/4863))
+- Fixing PIT flaky tests ([4632](https://github.com/opensearch-project/OpenSearch/pull/4632))
 -
 
 ### Security

buildSrc/src/integTest/java/org/opensearch/gradle/tar/SymbolicLinkPreservingTarIT.java

@@ -35,6 +35,7 @@
 import org.apache.commons.compress.archivers.tar.TarArchiveInputStream;
 import org.apache.commons.compress.compressors.bzip2.BZip2CompressorInputStream;
 import org.apache.commons.compress.compressors.gzip.GzipCompressorInputStream;
+import org.apache.tools.ant.taskdefs.condition.Os;
 import org.opensearch.gradle.test.GradleIntegrationTestCase;
 import org.gradle.api.GradleException;
 import org.gradle.testkit.runner.GradleRunner;
@@ -52,6 +53,7 @@
 
 import static org.hamcrest.CoreMatchers.anyOf;
 import static org.hamcrest.CoreMatchers.equalTo;
+import static org.junit.Assume.assumeFalse;
 
 public class SymbolicLinkPreservingTarIT extends GradleIntegrationTestCase {
 
@@ -60,6 +62,7 @@ public class SymbolicLinkPreservingTarIT extends GradleIntegrationTestCase {
 
     @Before
     public void before() throws IOException {
+        assumeFalse("Skip tar tests on windows.", Os.isFamily(Os.FAMILY_WINDOWS));
         final Path realFolder = temporaryFolder.getRoot().toPath().resolve("real-folder");
         Files.createDirectory(realFolder);
         Files.createFile(realFolder.resolve("file"));

buildSrc/src/main/java/org/opensearch/gradle/test/DistroTestPlugin.java

@@ -75,9 +75,9 @@
 import java.util.stream.Stream;
 
 public class DistroTestPlugin implements Plugin<Project> {
-    private static final String SYSTEM_JDK_VERSION = "11.0.16+8";
+    private static final String SYSTEM_JDK_VERSION = "11.0.17+8";
     private static final String SYSTEM_JDK_VENDOR = "adoptium";
-    private static final String GRADLE_JDK_VERSION = "17.0.4+8";
+    private static final String GRADLE_JDK_VERSION = "17.0.5+8";
     private static final String GRADLE_JDK_VENDOR = "adoptium";
 
     // all distributions used by distro tests. this is temporary until tests are per distribution

buildSrc/src/testFixtures/java/org/opensearch/gradle/test/JUnit3MethodProvider.java

@@ -59,7 +59,7 @@ public Collection<Method> getTestMethods(Class<?> suiteClass, ClassModel classMo
             if (m.getName().startsWith("test")
                 && Modifier.isPublic(m.getModifiers())
                 && !Modifier.isStatic(m.getModifiers())
-                && m.getParameterTypes().length == 0) {
+                && m.getParameterCount() == 0) {
                 result.add(m);
             }
         }

buildSrc/version.properties

@@ -2,7 +2,7 @@ opensearch        = 2.5.0
 lucene            = 9.4.1
 
 bundled_jdk_vendor = adoptium
-bundled_jdk = 17.0.4+8
+bundled_jdk = 17.0.5+8
 
 
 

client/rest-high-level/src/test/java/org/opensearch/client/PitIT.java

@@ -24,6 +24,7 @@
 import java.io.IOException;
 import java.util.ArrayList;
 import java.util.List;
+import java.util.concurrent.CountDownLatch;
 import java.util.concurrent.TimeUnit;
 import java.util.stream.Collectors;
 
@@ -71,7 +72,7 @@ public void testCreateAndDeletePit() throws IOException {
         assertTrue(deletePitResponse.getDeletePitResults().get(0).getPitId().equals(createPitResponse.getId()));
     }
 
-    public void testDeleteAllAndListAllPits() throws IOException {
+    public void testDeleteAllAndListAllPits() throws IOException, InterruptedException {
         CreatePitRequest pitRequest = new CreatePitRequest(new TimeValue(1, TimeUnit.DAYS), true, "index");
         CreatePitResponse pitResponse = execute(pitRequest, highLevelClient()::createPit, highLevelClient()::createPitAsync);
         CreatePitResponse pitResponse1 = execute(pitRequest, highLevelClient()::createPit, highLevelClient()::createPitAsync);
@@ -90,16 +91,19 @@ public void testDeleteAllAndListAllPits() throws IOException {
         List<String> pits = getAllPitResponse.getPitInfos().stream().map(r -> r.getPitId()).collect(Collectors.toList());
         assertTrue(pits.contains(pitResponse.getId()));
         assertTrue(pits.contains(pitResponse1.getId()));
+        CountDownLatch countDownLatch = new CountDownLatch(1);
         ActionListener<DeletePitResponse> deletePitListener = new ActionListener<>() {
             @Override
             public void onResponse(DeletePitResponse response) {
+                countDownLatch.countDown();
                 for (DeletePitInfo deletePitInfo : response.getDeletePitResults()) {
                     assertTrue(deletePitInfo.isSuccessful());
                 }
             }
 
             @Override
             public void onFailure(Exception e) {
+                countDownLatch.countDown();
                 if (!(e instanceof OpenSearchStatusException)) {
                     throw new AssertionError("Delete all failed");
                 }
@@ -123,6 +127,7 @@ public void onFailure(Exception e) {
         };
         highLevelClient().getAllPitsAsync(RequestOptions.DEFAULT, getPitsListener);
         highLevelClient().deleteAllPitsAsync(RequestOptions.DEFAULT, deletePitListener);
+        assertTrue(countDownLatch.await(10, TimeUnit.SECONDS));
         // validate no pits case
         getAllPitResponse = highLevelClient().getAllPits(RequestOptions.DEFAULT);
         assertTrue(getAllPitResponse.getPitInfos().size() == 0);

client/rest-high-level/src/test/java/org/opensearch/client/RestHighLevelClientTests.java

@@ -891,7 +891,10 @@ public void testApiNamingConventions() throws Exception {
             "remote_store.restore",
             "cluster.put_weighted_routing",
             "cluster.get_weighted_routing",
-            "cluster.delete_weighted_routing", };
+            "cluster.delete_weighted_routing",
+            "cluster.put_decommission_awareness",
+            "cluster.get_decommission_awareness",
+            "cluster.delete_decommission_awareness", };
         List<String> booleanReturnMethods = Arrays.asList("security.enable_user", "security.disable_user", "security.change_password");
         Set<String> deprecatedMethods = new HashSet<>();
         deprecatedMethods.add("indices.force_merge");
@@ -1004,37 +1007,34 @@ private static void assertSyncMethod(Method method, String apiName, List<String>
         }
 
         assertEquals("incorrect number of exceptions for method [" + method + "]", 1, method.getExceptionTypes().length);
+        final Class<?>[] parameterTypes = method.getParameterTypes();
         // a few methods don't accept a request object as argument
         if (APIS_WITHOUT_REQUEST_OBJECT.contains(apiName)) {
-            assertEquals("incorrect number of arguments for method [" + method + "]", 1, method.getParameterTypes().length);
-            assertThat(
-                "the parameter to method [" + method + "] is the wrong type",
-                method.getParameterTypes()[0],
-                equalTo(RequestOptions.class)
-            );
+            assertEquals("incorrect number of arguments for method [" + method + "]", 1, method.getParameterCount());
+            assertThat("the parameter to method [" + method + "] is the wrong type", parameterTypes[0], equalTo(RequestOptions.class));
         } else {
-            assertEquals("incorrect number of arguments for method [" + method + "]", 2, method.getParameterTypes().length);
+            assertEquals("incorrect number of arguments for method [" + method + "]", 2, method.getParameterCount());
             // This is no longer true for all methods. Some methods can contain these 2 args backwards because of deprecation
-            if (method.getParameterTypes()[0].equals(RequestOptions.class)) {
+            if (parameterTypes[0].equals(RequestOptions.class)) {
                 assertThat(
                     "the first parameter to method [" + method + "] is the wrong type",
-                    method.getParameterTypes()[0],
+                    parameterTypes[0],
                     equalTo(RequestOptions.class)
                 );
                 assertThat(
                     "the second parameter to method [" + method + "] is the wrong type",
-                    method.getParameterTypes()[1].getSimpleName(),
+                    parameterTypes[1].getSimpleName(),
                     endsWith("Request")
                 );
             } else {
                 assertThat(
                     "the first parameter to method [" + method + "] is the wrong type",
-                    method.getParameterTypes()[0].getSimpleName(),
+                    parameterTypes[0].getSimpleName(),
                     endsWith("Request")
                 );
                 assertThat(
                     "the second parameter to method [" + method + "] is the wrong type",
-                    method.getParameterTypes()[1],
+                    parameterTypes[1],
                     equalTo(RequestOptions.class)
                 );
             }
@@ -1048,39 +1048,40 @@ private static void assertAsyncMethod(Map<String, Set<Method>> methods, Method m
         );
         assertThat("async method [" + method + "] should return Cancellable", method.getReturnType(), equalTo(Cancellable.class));
         assertEquals("async method [" + method + "] should not throw any exceptions", 0, method.getExceptionTypes().length);
+        final Class<?>[] parameterTypes = method.getParameterTypes();
         if (APIS_WITHOUT_REQUEST_OBJECT.contains(apiName.replaceAll("_async$", ""))) {
-            assertEquals(2, method.getParameterTypes().length);
-            assertThat(method.getParameterTypes()[0], equalTo(RequestOptions.class));
-            assertThat(method.getParameterTypes()[1], equalTo(ActionListener.class));
+            assertEquals(2, parameterTypes.length);
+            assertThat(parameterTypes[0], equalTo(RequestOptions.class));
+            assertThat(parameterTypes[1], equalTo(ActionListener.class));
         } else {
-            assertEquals("async method [" + method + "] has the wrong number of arguments", 3, method.getParameterTypes().length);
+            assertEquals("async method [" + method + "] has the wrong number of arguments", 3, method.getParameterCount());
             // This is no longer true for all methods. Some methods can contain these 2 args backwards because of deprecation
-            if (method.getParameterTypes()[0].equals(RequestOptions.class)) {
+            if (parameterTypes[0].equals(RequestOptions.class)) {
                 assertThat(
                     "the first parameter to async method [" + method + "] should be a request type",
-                    method.getParameterTypes()[0],
+                    parameterTypes[0],
                     equalTo(RequestOptions.class)
                 );
                 assertThat(
                     "the second parameter to async method [" + method + "] is the wrong type",
-                    method.getParameterTypes()[1].getSimpleName(),
+                    parameterTypes[1].getSimpleName(),
                     endsWith("Request")
                 );
             } else {
                 assertThat(
                     "the first parameter to async method [" + method + "] should be a request type",
-                    method.getParameterTypes()[0].getSimpleName(),
+                    parameterTypes[0].getSimpleName(),
                     endsWith("Request")
                 );
                 assertThat(
                     "the second parameter to async method [" + method + "] is the wrong type",
-                    method.getParameterTypes()[1],
+                    parameterTypes[1],
                     equalTo(RequestOptions.class)
                 );
             }
             assertThat(
                 "the third parameter to async method [" + method + "] is the wrong type",
-                method.getParameterTypes()[2],
+                parameterTypes[2],
                 equalTo(ActionListener.class)
             );
         }
@@ -1093,16 +1094,17 @@ private static void assertSubmitTaskMethod(
         ClientYamlSuiteRestSpec restSpec
     ) {
         String methodName = extractMethodName(apiName);
+        final Class<?>[] parameterTypes = method.getParameterTypes();
         assertTrue("submit task method [" + method.getName() + "] doesn't have corresponding sync method", methods.containsKey(methodName));
-        assertEquals("submit task method [" + method + "] has the wrong number of arguments", 2, method.getParameterTypes().length);
+        assertEquals("submit task method [" + method + "] has the wrong number of arguments", 2, method.getParameterCount());
         assertThat(
             "the first parameter to submit task method [" + method + "] is the wrong type",
-            method.getParameterTypes()[0].getSimpleName(),
+            parameterTypes[0].getSimpleName(),
             endsWith("Request")
         );
         assertThat(
             "the second parameter to submit task method [" + method + "] is the wrong type",
-            method.getParameterTypes()[1],
+            parameterTypes[1],
             equalTo(RequestOptions.class)
         );
 

modules/lang-painless/src/main/java/org/opensearch/painless/Compiler.java

@@ -212,8 +212,9 @@ private static void addFactoryMethod(Map<String, Class<?>> additionalClasses, Cl
         }
 
         additionalClasses.put(factoryClass.getName(), factoryClass);
-        for (int i = 0; i < factoryMethod.getParameterTypes().length; ++i) {
-            Class<?> parameterClazz = factoryMethod.getParameterTypes()[i];
+        final Class<?>[] parameterTypes = factoryMethod.getParameterTypes();
+        for (int i = 0; i < parameterTypes.length; ++i) {
+            Class<?> parameterClazz = parameterTypes[i];
             additionalClasses.put(parameterClazz.getName(), parameterClazz);
         }
     }