Can we retrieve the matching tokens for a document?

[mrisher]

We're trying to display some highlighting in the UI based on which tokens match. I can't find any method that returns the matching tokens from a query, but wanted to confirm? Ideally we'd have something in the results object from the call results, scores = retriever.retrieve(...) so we would know that document ID #123 had tokens ["fish", "cat"]?

Thank you

[xhluca]

This is not built in, but i think it should be easy to verify? E.g.:

import bm25s

query = "what are cats?"
doc = "Cats are animals"

query_tokens = set(bm25s.tokenize(query, return_ids=False)[0])
doc_tokens = bm25s.tokenize(doc, return_ids=False)[0]

matches = [1 if dt in query_tokens else 0 for dt in doc_tokens]

[xhluca]

Code will be more complex for other cases. however, it should be fast since oyu are only interested in the top-k instances for highlighting.

[mrisher]

Thank you. Does that also give me a way to get token positions? I would like to boost the score for exact phrase matches and don't really understand the internals of the token index.

[xhluca]

No, but I think it would be straightforward to build a scorer for reranking bm25s results. Since this would be modular and separate from bm25s, i think it would be more appropriate as a separate tool, perhaps in the utils module or in a separate library.