Recommended enterprise RAG settings

[JaimeArboleda]

Hello!

I have a question regarding the management of huge collections of documents for an enterprise RAG application. Let's say that you rely on a vector database like Qdrant (which of course is great!) for performing vector or hybrid search. Ok, this solves one problem, but there is still the problem of managing the collection of documents: storing them, adding metadata to them, performing CRUD operations on them (and updating/propagating changes in the vector database as a consequence)... And we also have to consider chunking -taking into account the structure of the documents if possible- in this whole setting.

So I am very curious, how do you manage all this? Do you have some component -like, say, Alfresco- for handling the collection of documents as a whole? And some intermediate NoSQL database like MongoDB for storing the chunks and their metadata? Or everything is handled by the vector database? And how many pipelines do you have?

Thanks in advance!

[azayarni]

You can store all the chunks and metadata along with vectors in Qdrant. You do not need additional tools for this. https://qdrant.tech/documentation/concepts/payload/