You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
@ashraq1455 encountered an error where if a document were sufficiently large enough the worker would shutdown. The suspected cause was due to our semantic_chunkers.StatisticalChunker encoding sentence embeddings and storing all of them in-memory at the same time for the chunking methodology to run.
The proposed solution here would be to add a “rolling window” of focus that embeds a maximum number of sentences at any one time. This fix should be applied to both ConsecutiveChunker and StatisticalChunker in the semantic_chunkers library.
The text was updated successfully, but these errors were encountered:
Large documents chunking memory error
@ashraq1455 encountered an error where if a document were sufficiently large enough the worker would shutdown. The suspected cause was due to our
semantic_chunkers.StatisticalChunker
encoding sentence embeddings and storing all of them in-memory at the same time for the chunking methodology to run.The proposed solution here would be to add a “rolling window” of focus that embeds a maximum number of sentences at any one time. This fix should be applied to both
ConsecutiveChunker
andStatisticalChunker
in thesemantic_chunkers
library.The text was updated successfully, but these errors were encountered: