Almost usable for serious Production use - With specific suggestions after detailed testing

[mechanicmuthu]

Extensively tested this with our internal company data. Pointed suggestions are:

Preprocessing

• OCR the PDFs for a truer representation of private document questioning for e.g. using OCRmyPDF. This will convert pictures and scans into texts which can be used for input for the next step.

Ingestion

• After semantic search, the previous 1 and next 2 chunks can be added to the context of LLM. For enabling this the optional parameter ids (chunk ids) provided by langchain can be used and stored in db. Many times, the required chunks are partial in context.

Searching

• Semantic search doesn't work with specific keywords not part of general vocabulary (like technical words or IDs). Some lexical search must be implemented to take care of this. Something like Rank_BM25 can be used and the chunks which contain the lexican search result should be added to the input of LLM.

Presentation

• Users find most helpful the context of the references. Rather than showing them at last, show them first the references by calling .get_relevant_documents(query) on the retriever. In the meantime, our LLM can work on answering based on selected contexts.

Upgraded Logic

• Ask LLM first to provide different versions of the question and provide keywords for Lexical Search and use it as the input to Semantic similarity search (user usually have sub optimal questions) and then use the LLM again to answer based on Contexts the original question.

These will help this become considered for serious Production use.

[SilentKnightOwl]

All of these are fantastic suggestions, especially adding the previous 1 and next 2 chunks into the context. I have been using this for non-fiction books (epubs), and sometimes two or more of the references are just the title of a chapter.