How to apply bm25s to languages such as Chinese?

[AlanLu0808]

I think the bm25s library is great and very efficient. I would like to use it in my project.

But how to apply bm25s to languages such as Chinese? Can you provide some examples?

[bm777]

It works for Chinese language too. just use a Chinese tokenizer and stemmer.

[IeohMingChan]

After testing, it was found that due to the vocabulary in Chinese sentences is not connected by spaces and punctuation marks like in English, the BM25S tokenizer itself cannot separate the vocabulary in Chinese sentences. Therefore, Chinese sentences need to be pre splited, such as using jieba, then connect the separated vocabulary and input into the BM25S tokenizer.

[bm777]

btw BM25 tokenizer is directly coming from NLTK library.

bm25s/bm25s/tokenization.py

Lines 50 to 84 in 8b36bdd

	def _infer_stopwords(stopwords: Union[str, List[str]]) -> List[str]:
	# Source of stopwords: https://github.com/nltk/nltk/blob/96ee715997e1c8d9148b6d8e1b32f412f31c7ff7/nltk/corpus/__init__.py#L315
	if stopwords in ["english", "en", True]: # True is added to support the default
	return STOPWORDS_EN
	elif stopwords in ["english_plus", "en_plus"]:
	return STOPWORDS_EN_PLUS
	elif stopwords in ["german", "de"]:
	return STOPWORDS_GERMAN
	elif stopwords in ["dutch", "nl"]:
	return STOPWORDS_DUTCH
	elif stopwords in ["french", "fr"]:
	return STOPWORDS_FRENCH
	elif stopwords in ["spanish", "es"]:
	return STOPWORDS_SPANISH
	elif stopwords in ["portuguese", "pt"]:
	return STOPWORDS_PORTUGUESE
	elif stopwords in ["italian", "it"]:
	return STOPWORDS_ITALIAN
	elif stopwords in ["russian", "ru"]:
	return STOPWORDS_RUSSIAN
	elif stopwords in ["swedish", "sv"]:
	return STOPWORDS_SWEDISH
	elif stopwords in ["norwegian", "no"]:
	return STOPWORDS_NORWEGIAN
	elif stopwords in ["chinese", "zh"]:
	return STOPWORDS_CHINESE
	elif stopwords in [None, False]:
	return []
	elif isinstance(stopwords, str):
	raise ValueError(
	f"{stopwords} not recognized. Only English stopwords as default, German, Dutch, French, Spanish, Portuguese, Italian, Russian, Swedish, Norwegian, and Chinese are currently supported. "
	"Please input a list of stopwords"
	)
	else:
	return stopwords

[IeohMingChan]

Using Chinese stop words does not split Chinese sentences. Sentences are split based on token_pattern, and the regular expressions used by token_pattern cannot split Chinese sentences. I think Chinese stop words need to be used after splitting Chinese sentences.

[bm777]

@xhluca what do you think?

[xhluca]

You can use the stopwords separately for now. In v0.2 you will have a tokenizer class that can take a splitter function as input, so you can pass jieba function to it

[bm777]

@AlanLu0808 have a look here: #33

[xansar]

@AlanLu0808看看这里：#33

The tokenize function needs a 'token_pattern' to split words, which may not work for chinese. Using a split func as the input parameter may be better.