Add IVF changes to support Faiss byte vector #2002
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
naveentatikonda
added
Features
naveentatikonda
requested review from
heemin32,
navneet1v,
VijayanB,
vamshin,
jmazanec15,
junqiu-lei,
martin-gaievski,
ryanbogan and
luyuncheng
as code owners
naveentatikonda
force-pushed
the
add-support-faiss-byte-vector-ivf
branch
from
17a4e8b
to
60f21fb
Compare
Signed-off-by: Naveen Tatikonda <navtat@amazon.com>
Signed-off-by: Naveen Tatikonda <navtat@amazon.com>
naveentatikonda
force-pushed
the
add-support-faiss-byte-vector-ivf
branch
2 times, most recently
from
e4fd8c6
to
3b0c8da
Compare
Signed-off-by: Naveen Tatikonda <navtat@amazon.com>
naveentatikonda
force-pushed
the
add-support-faiss-byte-vector-ivf
branch
from
3b0c8da
to
121912f
Compare
navneet1v
reviewed
Aug 27, 2024
| @@ -320,6 +320,96 @@ void knn_jni::faiss_wrapper::CreateBinaryIndexFromTemplate(knn_jni::JNIUtilInter | |||
| faiss::write_index_binary(&idMap, indexPathCpp.c_str()); | |||
| } | |||
|
|
|||
| void knn_jni::faiss_wrapper::CreateByteIndexFromTemplate(knn_jni::JNIUtilInterface * jniUtil, JNIEnv * env, jintArray idsJ, | |||
There was a problem hiding this comment.
should we move these things to IndexService just like we have for other indices?
There was a problem hiding this comment.
Yes, we need to refactor these methods and move to IndexService. But, including those refactoring changes into this PR will make it big so want to do it later. Junqiu already started refactoring train Indices in #1918. Will collab with Junqiu and add my changes to that PR
src/main/java/org/opensearch/knn/training/BinaryTrainingDataConsumer.java
Outdated
Show resolved
Hide resolved
|
@naveentatikonda can we put a small benchmark run for this to showcase that ByteVector with IVF is working and we are seeing drop in memory usage. |
This is the RSS metrics for Cohere-wiki-768 dataset (475,858 vectors) and the memory usage is as expected. 
|
Signed-off-by: Naveen Tatikonda <navtat@amazon.com>
junqiu-lei
reviewed
Aug 30, 2024
opensearch-trigger-bot bot
pushed a commit
that referenced
this pull request
Aug 30, 2024
* Add HNSW changes to support Faiss byte vector Signed-off-by: Naveen Tatikonda <navtat@amazon.com> * Address Review Comments Signed-off-by: Naveen Tatikonda <navtat@amazon.com> * Add IVF changes to support Faiss byte vector Signed-off-by: Naveen Tatikonda <navtat@amazon.com> * Address Review Comments Signed-off-by: Naveen Tatikonda <navtat@amazon.com> --------- Signed-off-by: Naveen Tatikonda <navtat@amazon.com> (cherry picked from commit 2b303d9)
naveentatikonda
added a commit
that referenced
this pull request
Aug 30, 2024
* Add HNSW changes to support Faiss byte vector Signed-off-by: Naveen Tatikonda <navtat@amazon.com> * Address Review Comments Signed-off-by: Naveen Tatikonda <navtat@amazon.com> * Add IVF changes to support Faiss byte vector Signed-off-by: Naveen Tatikonda <navtat@amazon.com> * Address Review Comments Signed-off-by: Naveen Tatikonda <navtat@amazon.com> --------- Signed-off-by: Naveen Tatikonda <navtat@amazon.com> (cherry picked from commit 2b303d9) Co-authored-by: Naveen Tatikonda <navtat@amazon.com>
akashsha1
pushed a commit
to akashsha1/k-NN
that referenced
this pull request
Sep 16, 2024
* Add HNSW changes to support Faiss byte vector Signed-off-by: Naveen Tatikonda <navtat@amazon.com> * Address Review Comments Signed-off-by: Naveen Tatikonda <navtat@amazon.com> * Add IVF changes to support Faiss byte vector Signed-off-by: Naveen Tatikonda <navtat@amazon.com> * Address Review Comments Signed-off-by: Naveen Tatikonda <navtat@amazon.com> --------- Signed-off-by: Naveen Tatikonda <navtat@amazon.com> Signed-off-by: Akash Shankaran <akash.shankaran@intel.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Add IVF changes to support Faiss byte vector which behind the scenes uses Faiss SQ8_direct_signed scalar quantizer.
Related Issues
#1659
Check List
--signoff.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.