Issue with Updated Source Documents Not Being Recognized

[miguellacorte]

Hello,

I've been using the "privateGPT" tool and encountered an issue with updated source documents not being recognized. I added a new text file to the "source_documents" folder, but even after running the "ingest.py" and "privateGPT.py" scripts again, the tool continues to provide answers based on the old state of the union text that I previously deleted.

I've double-checked the placement and naming of the new text file, but it doesn't seem to have any effect. Could you please assist me in resolving this issue?

I would appreciate any guidance on how to ensure that the "privateGPT" tool properly ingests the updated source documents and generates answers based on the latest text.

Thank you for your support!

[imartinez]

The accumulation of documents is not working properly in the main branch. We are reviewing a PR to make it work properly:
#287

[vishnuvardhanvaka]

Yeah' I have solve it by manually changing the all_files in ingest.py folder to read file by file. Actually the glob isn't reading the files.If you want I can contribute my code.Just can the all_filesto as following code....
files=os.listdir(source_documents)
all_files=[source_dir+'/'+file for file in files]
You can remove the glob.glob and that all_files and replace with i suggested!

[imartinez]

Documents loading was fixed yesterday, but is still using glob. There is an open PR to move away from it, but I haven't experienced any issues with reading documents from the directory so far. Will take a closer look.

[PulpCattel]

@vishnuvardhanvaka @miguellacorte are you referring to this #327 (comment)?

Currently we are silently ignoring all unknown/missing file type. If that's your case, add the proper extension to your file so that it can be recognized, or wait until it's patched.