Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(primitive/faiss.py): support HNSW and reverted indexer #387

Merged
merged 8 commits into from
Sep 23, 2024

Conversation

tpoisonooo
Copy link
Collaborator

@tpoisonooo tpoisonooo commented Sep 12, 2024

Result

Query: "中秆早和原丰早,有啥区别?"

After

reverted-indexer gives more candidates
image

Usage

  1. prepare named entity list, for example words.txt
# words.txt
[
    "C两优2号",
    "大白谷",
    "中秆早",
    "原丰早"
]
  1. when building feature store, specify your words.txt
python3 -m huixiangdou.service.feature_store --ner-file words/words.json

After finish, there is a db_reverted_index in your workdir
image

  1. Test it as before
# command mode
python3 -m huixiangdou.main --standalone
# server mode
python3 -m huixiangdou.server
# gradio mode
..
# or evaluation mode whatever
python3 evaluation/end2end/main.py

@tpoisonooo tpoisonooo changed the title WIP: feat(primitive/faiss.py): support HNSW WIP: feat(primitive/faiss.py): support HNSW and reverted indexer Sep 19, 2024
@tpoisonooo tpoisonooo changed the title WIP: feat(primitive/faiss.py): support HNSW and reverted indexer feat(primitive/faiss.py): support HNSW and reverted indexer Sep 23, 2024
@tpoisonooo tpoisonooo merged commit 7e1be3f into main Sep 23, 2024
1 check passed
@tpoisonooo tpoisonooo deleted the support-cluster branch September 23, 2024 03:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant