Splitting/Chunking

[rdhillbb]

I am not sure this is the right forum to ask a question.
I have a very large document. I get the following message when I split the text.
Created a chunk of size 1870, which is longer than the specified 1000
text_splitter = CharacterTextSplitter(
separator="\n",
chunk_size=1000,
chunk_overlap=200,
length_function=len
)
texts = text_splitter.split_text(raw_text)

[Hamas-ur-Rehman]

you can try changing the separator to split on something else instead of line breaks or using a better splitter than this which can be like this one
text_splitter = RecursiveCharacterTextSplitter(
chunk_size=1000, chunk_overlap=0, separators=[" ", ",", "\n"]
)

[rdhillbb]

Thank you

[g-i-o-r-g-i-o]

I'm currently using this code, thanks for the suggestion. Don't know if has been supersed by something else meanwhile :-)

with open("input.txt") as f:
    text = f.read()
    text = text[:5000] # limit for testing

from langchain.text_splitter import RecursiveCharacterTextSplitter

# text_splitter = CharacterTextSplitter.from_tiktoken_encoder(
text_splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
    'gpt2', chunk_size=100, chunk_overlap=0
)
# separators=[" ", ",", "\n"]

texts = text_splitter.split_text(text)

len(texts), len(texts[0]), texts[:5]

[slobentanzer]

Hi @Hamas-ur-Rehman,
maybe this is a noob question, but why do you split on space, comma, and newline, but not on a stop (.)? Is there some resource that explains the technical details of text splitting?

[Hamas-ur-Rehman]

Hi Well from what I know, it seems that most sentences in a paragraph have usually many . full-stop characters, and the new line is usually used to tell paragraphs apart. So it is beneficial to split paragraphs as the meaning can be retained if we start splitting on full-stop characters we may get unintended splits and usually sentences depend on each other so this might not be a better approch.

[slobentanzer]

Thanks! Makes sense that sentences depend on each other, but i am not sure i get why a sentence would usually have many full-stops. Do you have an example?

For my use case (in-context learning on scientific literature), it could be desirable to maybe even have an option to embed multiple sentences, but with an overlap (say, sentence 1-2-3, sentence 2-3-4, sentence 3-4-5, and so on). Are you aware of an option to do that? Or is there some reason this does not make sense / is not necessary when using the RecursiveCharacterTextSplitter?

Also happy to read for myself if you can point me to a resource. :)

[Hamas-ur-Rehman]

Here are some examples of what multiple full-stops may look like

1- "The U.S. is located in North America. Its capital city is Washington, D.C. and it has a population of over 300 million people."
2- "I had to visit three different cities yesterday: New York, NY., Philadelphia, PA., and Boston, MA."
3- "I'm not a big fan of public speaking. In fact, I get very nervous and my palms start to sweat. However, I know it's an important skill to have." 

and you don't necessarily have to use this these text splitters you can define your own like this

Here is a Python function that splits text into multiple chunks, where each chunk contains a specified number of sentences with an overlap of a specified number of sentences:

import nltk

def split_text_with_overlap(text, num_sentences=3, overlap=1):
    """
    Splits the input text into multiple chunks, where each chunk contains
    a specified number of sentences with an overlap of a specified number
    of sentences.

    Args:
    - text (str): The input text to be split.
    - num_sentences (int): The number of sentences to include in each chunk.
      Default is 3.
    - overlap (int): The number of sentences to overlap between adjacent
      chunks. Default is 1.

    Returns:
    - A list of text chunks, where each chunk contains a specified number
      of sentences with an overlap of a specified number of sentences.
    """

    # Tokenize the text into sentences using NLTK
    sentences = nltk.sent_tokenize(text)

    # Initialize the list of text chunks
    chunks = []

    # Split the sentences into chunks with the specified overlap
    start_idx = 0
    while start_idx < len(sentences):
        end_idx = min(start_idx + num_sentences, len(sentences))
        chunk = ' '.join(sentences[start_idx:end_idx])
        chunks.append(chunk)
        start_idx += num_sentences - overlap

    return chunks

Here's an example of how to use this function with a sample text:

text = "This is a sample text. It contains multiple sentences. I want to split it into chunks. Each chunk should contain 2 sentences with an overlap of 1 sentence."
chunks = split_text_with_overlap(text, num_sentences=2, overlap=1)
print(chunks)

Output:

['This is a sample text. It contains multiple sentences.',
 'It contains multiple sentences. I want to split it into chunks.',
 'I want to split it into chunks. Each chunk should contain 2 sentences.',
 'Each chunk should contain 2 sentences with an overlap of 1 sentence.']

In this example, the split_text_with_overlap function splits the input text into chunks of 2 sentences each, with an overlap of 1 sentence between adjacent chunks. The resulting chunks are printed to the console.

The split-code function was generated with ChatGPT

And what you can do is load it to your vector db so like if you are loading to chroma db you can

vectordb = Chroma(embedding_function=embedding)
for t in chunks:
    Chroma.add_texts(vectordb,texts=[t])
vectordb.persist()

[slobentanzer]

I see what you mean, thanks for the comprehensive answer, @Hamas-ur-Rehman! :)

[chaolili]

You could use TokenTextSplitter instead of CharacterTextSplitter, which could accurately split text based on the chunk_size, with each chunk having the same token

[slobentanzer]

also a good idea, thanks for the input :)

[GagandeepDulku]

If Is there any way to stop this message from printing Created a chunk of size n, which is longer than the specified x, something like verbose = False somewhere?

[jacksonjack001]

come from gp4-preview

import logging
logging.getLogger().setLevel(logging.ERROR)