[Remote Vector Index Build] Initial implementation of Index Build Service #2532
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…t#2442) * move build to one directory, fix broken flag Signed-off-by: Samuel Herman <sherman8915@gmail.com> * fix make path Signed-off-by: Samuel Herman <sherman8915@gmail.com> * changelog update Signed-off-by: Samuel Herman <sherman8915@gmail.com> * add fix for classpath change and for cmake discovery on macos Signed-off-by: Samuel Herman <sherman8915@gmail.com> * fix make discovery for gradle Signed-off-by: Samuel Herman <sherman8915@gmail.com> * fix cmake path for macOS Signed-off-by: Samuel Herman <sherman8915@gmail.com> --------- Signed-off-by: Samuel Herman <sherman8915@gmail.com>
* Update package name to fix compilation issue Core renamed this package in opensearch-project/OpenSearch#17272 This commit renames package accordingly Signed-off-by: Balasubramanian <balasvij@amazon.com> * Update build.gradle and build.sh to separate x64 linux nmslib build with different gcc versions (opensearch-project#2506) (opensearch-project#2508) * Update build gradle to separate nmslib / faiss generation Signed-off-by: Peter Zhu <zhujiaxi@amazon.com> * Update scripts/build.sh to separate enable gcc switch Signed-off-by: Peter Zhu <zhujiaxi@amazon.com> * Remove test comments Signed-off-by: Peter Zhu <zhujiaxi@amazon.com> * Remove test comments Signed-off-by: Peter Zhu <zhujiaxi@amazon.com> * Remove test comments Signed-off-by: Peter Zhu <zhujiaxi@amazon.com> * Updating restart and rolling upgrade bwc test bundle.gradle Signed-off-by: Peter Zhu <zhujiaxi@amazon.com> * Enforce gcc10 for nmslib to compile and avx512_spr have no-op Signed-off-by: Peter Zhu <zhujiaxi@amazon.com> --------- Signed-off-by: Peter Zhu <zhujiaxi@amazon.com> (cherry picked from commit 107c4f1) Co-authored-by: Peter Zhu <zhujiaxi@amazon.com> --------- Signed-off-by: Balasubramanian <balasvij@amazon.com> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> Co-authored-by: Peter Zhu <zhujiaxi@amazon.com>
…pensearch-project#2470) * Adds debug logs for KNNQuery and KNNWeight Signed-off-by: Tejas Shah <shatejas@amazon.com> * Adds check to see if log is enabled to start and stop StopWatch Signed-off-by: Tejas Shah <shatejas@amazon.com> * Addressing comments on the PR Signed-off-by: Tejas Shah <shatejas@amazon.com> * Adds shard and segment info in the logs Signed-off-by: Tejas Shah <shatejas@amazon.com> * Removes unnecessary segment name param from exact search Signed-off-by: Tejas Shah <shatejas@amazon.com> * Fixes the build Signed-off-by: Tejas Shah <shatejas@amazon.com> --------- Signed-off-by: Tejas Shah <shatejas@amazon.com> (cherry picked from commit f322e27) Co-authored-by: Tejas Shah <shatejas@amazon.com>
* clean JNI artifacts with ./gradlew clean Signed-off-by: Samuel Herman <sherman8915@gmail.com> * nest release under build directory Signed-off-by: Samuel Herman <sherman8915@gmail.com> * adjust all references to the old release path Signed-off-by: Samuel Herman <sherman8915@gmail.com> * remove irrelevant paths from gitignore, add jni/build Signed-off-by: Samuel Herman <sherman8915@gmail.com> * fix logging on linux Signed-off-by: Samuel Herman <sherman8915@gmail.com> --------- Signed-off-by: Samuel Herman <sherman8915@gmail.com>
* Remove skip building graph check for quantization use case (opensearch-project#2430) For quantization indices, we don't have to apply building graph check since it is already faster, this is now only applied for fp32/16 indices and where threshold is configured. Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com> * Update default to 0 to always build graph as default behavior (opensearch-project#2452) Signed-off-by: Balasubramanian <balasvij@amazon.com> * Update changelog Signed-off-by: Balasubramanian <balasvij@amazon.com> --------- Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com> Signed-off-by: Balasubramanian <balasvij@amazon.com>
* Add release notes for 2.19.0 Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> * Fix links for release notes Co-authored-by: John Mazanec <jmazane@amazon.com> Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> --------- Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> Co-authored-by: John Mazanec <jmazane@amazon.com>
…pensearch-project#2526) Signed-off-by: Peter Zhu <zhujiaxi@amazon.com>
…2528) Signed-off-by: Tejas Shah <shatejas@amazon.com>
Signed-off-by: Rohan Chitale <r.chitale7@gmail.com>
rchitale7
requested review from
heemin32,
navneet1v,
VijayanB,
vamshin,
jmazanec15,
naveentatikonda,
junqiu-lei,
martin-gaievski,
ryanbogan,
luyuncheng,
shatejas,
0ctopus13prime and
Vikasht34
as code owners
navneet1v
merged commit 91d66eb
into
opensearch-project:feature/remote-index-build-service
16 of 47 checks passed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR contains changes for the Remote Index Build Service. It is being merged into a feature branch for now, as it does not include unit tests, and the final result will be merged into a separate repository. These changes are for the 'control flow' of the service - follow up PRs will be raised for the object store and faiss client implementations.
Related Issues
TODO: Create GitHub Issue for Index Build Service Control Flow Design.
Check List
--signoff.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.