Learn from JSON files

[jlsajfj]

JSON is basically just JS / txt files and I'm kind of getting annoyed that I keep having to rename extension to JS to use (Maybe no limitation on files that aren't binaries?)

[JasonWeill]

@jlsajfj Thanks for opening this PR! Could you please open a feature request to modify /learn to learn from all readable text files in the chosen directory?

[dlqqq]

@jlsajfj The reason we omitted JSON from the list of file types accepted by /learn was because when this feature was initially developed nearly 2 years ago, there did not exist a way to split large JSON files to be embedded. However, there now exists a RecursiveJsonSplitter class in LangChain that can be used to split JSON files.

Can you modify packages/jupyter-ai/jupyter_ai/chat_handlers/learn.py to use this new splitter class? This would be a two step process:

1. Add the below import:

from langchain.text_splitter import (
    LatexTextSplitter,
    MarkdownTextSplitter,
    PythonCodeTextSplitter,
    RecursiveCharacterTextSplitter,
+   RecursiveJsonSplitter,
)

2. Add the below entry to the splitters dictionary in learn_dir():

    async def learn_dir(
        self, path: str, chunk_size: int, chunk_overlap: int, all_files: bool = False
    ):
        dask_client: DaskClient = await self.dask_client_future
        splitter_kwargs = {"chunk_size": chunk_size, "chunk_overlap": chunk_overlap}
        splitters = {
            ".py": PythonCodeTextSplitter(**splitter_kwargs),
            ".md": MarkdownTextSplitter(**splitter_kwargs),
            ".tex": LatexTextSplitter(**splitter_kwargs),
            ".ipynb": NotebookSplitter(**splitter_kwargs),
+           ".json": RecursiveJsonSplitter(**splitter_kwargs),
        }

Optionally, I can help make these changes if you give me permission to push to your fork's branch.

As an immediate workaround for getting context from JSON files, you can use the new context command @file:<file-path>. When you send a message with this command included, it will include the entire contents of the specified file with your prompt. This is already available in Jupyter AI v2.24.0 and above.

[dlqqq]

Thank you for your contribution! I can help implement my suggestion above in a follow-up PR, no worries.

[jlsajfj]

Oh shoot- I seem to not get emails about this for some reason... Thanks for the follow up + the fix as well! I'll have to figure out how to get notified properly...