add sense2vec support too and integrate with POS-config

[davidberenstein1957]

No description provided.

[prakhar251998]

Hi @davidberenstein1957, saw the capabailities of sense2vec library and how it will work much better than some of the pretrained glove word2vec models.
My question was is there are a way we can add support for more state of the art word vector embeddings like sentence transformers,BERT etc.?

[davidberenstein1957]

@prakhar251998 thanks for the suggestion, but sadly this wouldn´t be possible. The concise-concepts library works based on a find_most_similar search within pre-defined embeddings based on tokens present in the embedding model. For word2vec-like models, these tokens are pre-defined/indexed and have a stand-alone semantical meaning like apple being used in a similar context as pear. For transformer-based models, the index is mostly limited to a sub-word/character level and therefore doesn´t allow for a find_most_similar operation.

I you would like to use these kinds of embeddings, you could potentially create a semantic-search knowledge base with KNN/ANN and embeddings based on the descriptions of the potential entities, but maybe this costs too much effort.