Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Don't try to embed (recently) deleted documents #2618

Open
cebtenzzre opened this issue Jul 9, 2024 · 0 comments
Open

Don't try to embed (recently) deleted documents #2618

cebtenzzre opened this issue Jul 9, 2024 · 0 comments
Labels
chat gpt4all-chat issues chat-ui-ux Issues related to the look and feel of GPT4All Chat. enhancement New feature or request local-docs

Comments

@cebtenzzre
Copy link
Member

cebtenzzre commented Jul 9, 2024

Right now, the localdocs database queues chunks to be embedded via Qt signals to the embedding thread. There is no upper limit to the number of chunks that can be queued, and since the text is included, no check is done of whether the document still exists by the time it is finally popped from the queue.

Try adding a large file to a LocalDocs collection and then removing it shortly after. You will see a lot of CPU usage and LocalDocs showing partial progress for some time, despite no useful work being done.

It would not be hard to add a check on the receiving side of the queue to verify that the chunks are still in the database, so we do not perform unnecessary work.

@cebtenzzre cebtenzzre added enhancement New feature or request chat gpt4all-chat issues local-docs chat-ui-ux Issues related to the look and feel of GPT4All Chat. labels Jul 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
chat gpt4all-chat issues chat-ui-ux Issues related to the look and feel of GPT4All Chat. enhancement New feature or request local-docs
Projects
None yet
Development

No branches or pull requests

1 participant