-
Notifications
You must be signed in to change notification settings - Fork 7.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Provide new chunking strategies in localdocs #2635
Comments
Semantic Chunking in practice: https://boudhayan-dev.medium.com/semantic-chunking-in-practice-23a8bc33d56d I also think a very natural long term goal for GPT4All could be having responses based on Agents using knowledge graphs fed in via RAG using Nomic Maps (but that goes beyond a simple "chunking strategy"). |
is it at least possible to change easy embedding models ? |
Issue: #2635 Signed-off-by: Adam Treat <treat.adam@gmail.com>
Somebody went all in on RegEx lol
|
This is assuming the text is even structured properly. The pdf text we get right now does not have formatting really to regex on very much. |
Currently we do a character/word based chunking that is very simple. We should enhance our chunking strategies to possibly include:
Here is some possible literature:
The text was updated successfully, but these errors were encountered: