Update ld_library_path for k-NN in release Dockerfile

[jmazanec15]

Signed-off-by: John Mazanec jmazane@amazon.com

Description

For k-NN, we needed to package a dependency to openmp in knnlib build directory. This currently works for the tar install because we set LD_LIBRARY_PATH env variable to plugins/opensearch-knn/knnlib.

However, this does not work for the release docker image because instead of setting LD_LIBRARY_PATH, we copy the lib to /usr/lib. My first attempts to fix this tried to generalize this copy statement to also copy the libomp to /usr/lib. However, this was not working. On x64, it was only looking for the libomp in /usr/lib64 and on arm it was looking for k-NN lib only in /usr/lib.

To simplify, I decided to just replace copy with env set (as is done in the tar install).

To test, I built the image on AL2 arm64/x64 machines with:

./build-image-single-arch.sh -v 1.2.0 -f ./dockerfiles/opensearch.al2.dockerfile -p opensearch -a  x64 -t opensearch-1.2.0-linux-x64.tar.gz

Step 20/30 : RUN echo "export JAVA_HOME=$OPENSEARCH_HOME/jdk" >> /etc/profile.d/java_home.sh
 ---> Running in daa3e1307696
Removing intermediate container daa3e1307696
 ---> 29b77b9a68a0
Step 21/30 : ENV LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$OPENSEARCH_HOME/plugins/opensearch-knn/knnlib"
 ---> Running in 2f5312236439
Removing intermediate container 2f5312236439
 ---> f8c0b2868d59
Step 22/30 : USER $UID
 ---> Running in aa426ef800aa
Removing intermediate container aa426ef800aa

sudo ./build-image-single-arch.sh -v 1.2.0 -f ./dockerfiles/opensearch.al2.dockerfile -p opensearch -a arm64 -t opensearch-1.2.0-linux-arm64.tar.gz

Step 20/30 : RUN echo "export JAVA_HOME=$OPENSEARCH_HOME/jdk" >> /etc/profile.d/java_home.sh
 ---> Running in 9d816ef75478
Removing intermediate container 9d816ef75478
 ---> af86fcd17db1
Step 21/30 : ENV LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$OPENSEARCH_HOME/plugins/opensearch-knn/knnlib"
 ---> Running in 796dee275ca4
Removing intermediate container 796dee275ca4
 ---> 8694b2004df3
Step 22/30 : USER $UID
 ---> Running in b7f15fa57fb0
Removing intermediate container b7f15fa57fb0

I then created a bash session container with the images, ran opensearch-docker-entrypoint.sh, and confirmed knn libs load as expected when knn workload was run. Note - I was running into unrelated problems with security/perf analyzer, so I disabled them in my test.

The tars I used came from build 867, which incorporated opensearch-project/k-NN#175.

Issues Resolved

opensearch-project/k-NN#174

Check List

• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[peterzhuamazon]

Have you test this with arm64 as well? Thanks.

[codecov-commenter]

Codecov Report

Merging #879 (0d1be5f) into main (e31441f) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #879   +/-   ##
=======================================
  Coverage   91.22%   91.22%           
=======================================
  Files          88       88           
  Lines        2382     2382           
=======================================
  Hits         2173     2173           
  Misses        209      209           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 11cff07...0d1be5f. Read the comment docs.

[jmazanec15]

@peterzhuamazon yes, I built image on AL2 arm64 and x64 machines and ran k-NN workflow.

[dblock]

This works. I wonder whether there's a generic mechanism we could introduce into OpenSearch core whereas plugins/anything/libs is added to LD_LIBRARY_PATH automatically? Or added at plugin load time (cc @saratvemulapalli)? Would also be part of opensearch-project/opensearch-plugins#91