Move ImmutableOpenMap usage to j.u.Map and upgrade gradle to 7.6.1 #885

jmazanec15 · 2023-05-02T22:20:13Z

Description

Core deleted ImmutableOpenMap in opensearch-project/OpenSearch#7309. PR removes usages of this class. Additionally, build is failing due to gradle 7.5 issue - PR changes gradle to 7.6.1 as was done in #878. Lastly, updating BWC tests for SNAPSHOT-2.8.

Gradle was updated with following command:

gradle wrapper --gradle-version 7.6.1

Check List

Commits are signed as per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: John Mazanec <jmazane@amazon.com>

codecov-commenter · 2023-05-02T22:37:08Z

Codecov Report

Merging #885 (7a6565d) into main (fd4b9f1) will decrease coverage by 0.12%.
The diff coverage is 100.00%.

❗ Current head 7a6565d differs from pull request most recent head 9c977d7. Consider uploading reports for the commit 9c977d7 to get more accurate results

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

@@             Coverage Diff              @@
##               main     #885      +/-   ##
============================================
- Coverage     85.35%   85.24%   -0.12%     
+ Complexity     1089     1087       -2     
============================================
  Files           152      152              
  Lines          4404     4404              
  Branches        389      389              
============================================
- Hits           3759     3754       -5     
- Misses          470      473       +3     
- Partials        175      177       +2

Impacted Files	Coverage Δ
...in/transport/TrainingJobRouterTransportAction.java	`80.00% <100.00%> (ø)`

... and 1 file with indirect coverage changes

Signed-off-by: John Mazanec <jmazane@amazon.com>

jmazanec15 · 2023-05-04T04:29:07Z

Secure test is failing:

java.lang.NoSuchMethodError: 'java.util.Map org.opensearch.cluster.node.DiscoveryNodes.getDataNodes()'
        at org.opensearch.knn.plugin.transport.TrainingJobRouterTransportAction.selectNode(TrainingJobRouterTransportAction.java:99)
        at org.opensearch.knn.plugin.transport.TrainingJobRouterTransportAction.lambda$routeRequest$1(TrainingJobRouterTransportAction.java:75)
        at org.opensearch.action.ActionListener$1.onResponse(ActionListener.java:80)
        at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:113)
        at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:107)
        at org.opensearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:73)
        at org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:88)
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
        at org.opensearch.common.util.concurrent.OpenSearchExecutors$DirectExecutorService.execute(OpenSearchExecutors.java:341)
        at org.opensearch.action.support.nodes.TransportNodesAction$AsyncAction.finishHim(TransportNodesAction.java:315)
        at org.opensearch.action.support.nodes.TransportNodesAction$AsyncAction.onOperation(TransportNodesAction.java:300)
        at org.opensearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleResponse(TransportNodesAction.java:277)
        at org.opensearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleResponse(TransportNodesAction.java:269)
        at org.opensearch.transport.TransportService$6.handleResponse(TransportService.java:777)
        at org.opensearch.security.transport.SecurityInterceptor$RestoringTransportResponseHandler.handleResponse(SecurityInterceptor.java:306)
        at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1381)
        at org.opensearch.transport.TransportService$DirectResponseChannel.processResponse(TransportService.java:1464)
        at org.opensearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1444)
        at org.opensearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:70)
        at org.opensearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:328)
        at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceivedDecorate(SecuritySSLRequestHandler.java:192)
        at org.opensearch.security.transport.SecurityRequestHandler.messageReceivedDecorate(SecurityRequestHandler.java:170)
        at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceived(SecuritySSLRequestHandler.java:96)
        at org.opensearch.security.OpenSearchSecurityPlugin$7$1.messageReceived(OpenSearchSecurityPlugin.java:675)
        at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:106)
        at org.opensearch.transport.TransportService$8.doRun(TransportService.java:973)
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:806)
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1589)

This is because 3.0.0 staging docker image is old and doesnt incorporate the ImmutableOpenMap change: https://hub.docker.com/layers/opensearchstaging/opensearch/3.0.0/images/sha256-3f3dbc965fcaf8627a4b35996e57548aedf346083fb220484878f93412759e11?context=explore

martin-gaievski · 2023-05-04T23:20:27Z

src/main/java/org/opensearch/knn/plugin/transport/TrainingJobRouterTransportAction.java

@@ -21,13 +21,15 @@
 import org.opensearch.cluster.service.ClusterService;
 import org.opensearch.common.Strings;
 import org.opensearch.common.ValidationException;
-import org.opensearch.common.collect.ImmutableOpenMap;
+// import org.opensearch.common.collect.ImmutableOpenMap;


seems this line is not needed

martin-gaievski · 2023-05-04T23:23:00Z

gradle/wrapper/gradle-wrapper.properties

-#  Copyright OpenSearch Contributors
-#  SPDX-License-Identifier: Apache-2.0
-#
-


I think we still need to have a license header, please check core repo, they kept it after gradle 8.1.1 upgrade https://github.com/opensearch-project/OpenSearch/blob/main/gradle/wrapper/gradle-wrapper.properties

Signed-off-by: John Mazanec <jmazane@amazon.com>

heemin32 · 2023-05-04T23:26:43Z

gradle/wrapper/gradle-wrapper.properties

 zipStoreBase=GRADLE_USER_HOME
 zipStorePath=wrapper/dists
-distributionSha256Sum=97a52d145762adc241bad7fd18289bf7f6801e08ece6badf80402fe2b9f250b1


can we have sha256 for new version of gradle?

Good point. Updated.

Signed-off-by: John Mazanec <jmazane@amazon.com>

opensearch-trigger-bot · 2023-06-30T20:26:29Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.x 2.x
# Navigate to the new working tree
cd .worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-885-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 5a0b20d81b48874f995f8295919c62b5460602e4
# Push it to GitHub
git push --set-upstream origin backport/backport-885-to-2.x
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-885-to-2.x.

nknize · 2023-06-30T20:27:13Z

ImmutableOpenMap removal backport PR landed so this needs to be backported.

navneet1v · 2023-07-07T18:52:46Z

@junqiu-lei the release owner for 2.9. can we backport this PR to 2.x

opensearch-trigger-bot · 2023-07-07T18:53:16Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.x 2.x
# Navigate to the new working tree
cd .worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-885-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 5a0b20d81b48874f995f8295919c62b5460602e4
# Push it to GitHub
git push --set-upstream origin backport/backport-885-to-2.x
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-885-to-2.x.

opensearch-trigger-bot · 2023-07-07T19:33:05Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.x 2.x
# Navigate to the new working tree
cd .worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-885-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 5a0b20d81b48874f995f8295919c62b5460602e4
# Push it to GitHub
git push --set-upstream origin backport/backport-885-to-2.x
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-885-to-2.x.

jmazanec15 added 2 commits May 2, 2023 15:09

Upgrade gradle to 7.6.1

77db96a

Signed-off-by: John Mazanec <jmazane@amazon.com>

Switch away from OpenSearch internal map

075093f

Signed-off-by: John Mazanec <jmazane@amazon.com>

jmazanec15 added the skip-changelog label May 2, 2023

Update backwards compatibility for 2.7 release

c412158

Signed-off-by: John Mazanec <jmazane@amazon.com>

This was referenced May 4, 2023

Enable knn in 2.8 opensearch-project/opensearch-build#3472

Merged

Move old manifests to legacy opensearch-project/opensearch-build#3473

Merged

martin-gaievski reviewed May 4, 2023

View reviewed changes

remove unused import

3927aab

Signed-off-by: John Mazanec <jmazane@amazon.com>

jmazanec15 requested review from heemin32, navneet1v, VijayanB, vamshin, naveentatikonda and junqiu-lei as code owners May 4, 2023 23:23

Update license

7a6565d

Signed-off-by: John Mazanec <jmazane@amazon.com>

jmazanec15 requested a review from martin-gaievski May 4, 2023 23:26

heemin32 reviewed May 4, 2023

View reviewed changes

Add gradle checksum

9c977d7

Signed-off-by: John Mazanec <jmazane@amazon.com>

jmazanec15 requested a review from heemin32 May 4, 2023 23:45

martin-gaievski approved these changes May 4, 2023

View reviewed changes

heemin32 approved these changes May 5, 2023

View reviewed changes

jmazanec15 merged commit 5a0b20d into opensearch-project:main May 5, 2023

b4sjoo mentioned this pull request May 11, 2023

[BUG] ML-Commons Plugin build failure due to OpenSearch Core breaking changes opensearch-project/ml-commons#891

Closed

nknize added the backport 2.x label Jun 30, 2023

navneet1v added backport 2.x and removed backport 2.x labels Jul 7, 2023

junqiu-lei added backport 2.x and removed backport 2.x labels Jul 7, 2023

This was referenced Jul 7, 2023

[Backport 2.x] Move ImmutableOpenMap usage to j.u.Map and upgrade gradle to 7.6.1 #954

Closed

Move ImmutableOpenMap usage to j.u.Map #955

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move ImmutableOpenMap usage to j.u.Map and upgrade gradle to 7.6.1 #885

Move ImmutableOpenMap usage to j.u.Map and upgrade gradle to 7.6.1 #885

jmazanec15 commented May 2, 2023 •

edited

Loading

codecov-commenter commented May 2, 2023 •

edited

Loading

jmazanec15 commented May 4, 2023

martin-gaievski May 4, 2023

jmazanec15 May 4, 2023

martin-gaievski May 4, 2023

jmazanec15 May 4, 2023

heemin32 May 4, 2023

jmazanec15 May 4, 2023

opensearch-trigger-bot bot commented Jun 30, 2023

nknize commented Jun 30, 2023

navneet1v commented Jul 7, 2023

opensearch-trigger-bot bot commented Jul 7, 2023

opensearch-trigger-bot bot commented Jul 7, 2023

Move ImmutableOpenMap usage to j.u.Map and upgrade gradle to 7.6.1 #885

Move ImmutableOpenMap usage to j.u.Map and upgrade gradle to 7.6.1 #885

Conversation

jmazanec15 commented May 2, 2023 • edited Loading

Description

Check List

codecov-commenter commented May 2, 2023 • edited Loading

Codecov Report

jmazanec15 commented May 4, 2023

martin-gaievski May 4, 2023

Choose a reason for hiding this comment

jmazanec15 May 4, 2023

Choose a reason for hiding this comment

martin-gaievski May 4, 2023

Choose a reason for hiding this comment

jmazanec15 May 4, 2023

Choose a reason for hiding this comment

heemin32 May 4, 2023

Choose a reason for hiding this comment

jmazanec15 May 4, 2023

Choose a reason for hiding this comment

opensearch-trigger-bot bot commented Jun 30, 2023

nknize commented Jun 30, 2023

navneet1v commented Jul 7, 2023

opensearch-trigger-bot bot commented Jul 7, 2023

opensearch-trigger-bot bot commented Jul 7, 2023

jmazanec15 commented May 2, 2023 •

edited

Loading

codecov-commenter commented May 2, 2023 •

edited

Loading