Refactor SelfQueryRetriever

[pprados]

The current implementation of SelfQueryRetriever relies on a list of Translators built in the function _get_builtin_translator(). This function depends on all known VectorStores and returns the corresponding translator.

The current implementation has maintainability and scalability issues. Indeed, when adding a new vector store, it is necessary to intervene in langchain-community and langchain to publish a new version. This is not aligned with the goal of modularizing into several components (See PR Fix SelfQuery with PGVector).

Another difficulty is that SelfQueryRetriever is present in langchain, but relies on Translators present in langchain-community.

We propose a new approach. Instead of having a mapping between VectorStore and Translator initialized by _get_builtin_translator(), we believe it is more pertinent to use a global dictionary where the different implementations can register their Translators themselves.

Thus, the code can be reorganized as follows:

• A global dictionary GLOBAL_BUILTIN_TRANSLATORS is proposed
• self_query/base.py provides def register_translator(vectorstore_cls: Type[VectorStore], translator_cls: Type[Visitor]) -> None
  (or registers a lambda with kwargs, to handle cases like QdrantTranslator(metadata_key=vectorstore.metadata_payload_key))
• SelfQueryRetriever migrates to langchain-core or langchain-community
• Each VectorStore has a mechanism to register its Translator

# at the end of pgvector.py
try:
    from langchain.retrievers.self_query.base import register_translator
    from langchain_community.query_constructors.pgvector import PGVectorTranslator

    register_translator(PGVector, PGVectorTranslator)
except ImportError:  # Due to the circular dependency, we cannot import SelfQueryRetriever
    pass  # pragma: no cover

The different Translators will be migrated to the respective projects within langchain-community or others, like langchain-postgres for example. Thus, the different Translators are only registered if a code imports the corresponding VectorStore.

What do you think?

[eyurtsev]

@pprados, @baskaryan and I are working through the base abstractions right now for search and indexing. We want the implementation of the vectorstore to expose a more standardized filtering interface as part of its API. This should make it unnecessary to have a mapping between vectorstores and translators.

[pprados]

I have a VectorStore wrapper class. I've already told you about it: RAGVectorStore. I had proposed a generic wrapper, to enable the filters associated with delegated vectorstores to be used.
I hope your next implementation will be compatible with this usage.