-
-
Notifications
You must be signed in to change notification settings - Fork 322
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Learn from JSON files #1024
Learn from JSON files #1024
Conversation
@jlsajfj Thanks for opening this PR! Could you please open a feature request to modify |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jlsajfj The reason we omitted JSON from the list of file types accepted by /learn
was because when this feature was initially developed nearly 2 years ago, there did not exist a way to split large JSON files to be embedded. However, there now exists a RecursiveJsonSplitter
class in LangChain that can be used to split JSON files.
Can you modify packages/jupyter-ai/jupyter_ai/chat_handlers/learn.py
to use this new splitter class? This would be a two step process:
- Add the below import:
from langchain.text_splitter import (
LatexTextSplitter,
MarkdownTextSplitter,
PythonCodeTextSplitter,
RecursiveCharacterTextSplitter,
+ RecursiveJsonSplitter,
)
- Add the below entry to the
splitters
dictionary inlearn_dir()
:
async def learn_dir(
self, path: str, chunk_size: int, chunk_overlap: int, all_files: bool = False
):
dask_client: DaskClient = await self.dask_client_future
splitter_kwargs = {"chunk_size": chunk_size, "chunk_overlap": chunk_overlap}
splitters = {
".py": PythonCodeTextSplitter(**splitter_kwargs),
".md": MarkdownTextSplitter(**splitter_kwargs),
".tex": LatexTextSplitter(**splitter_kwargs),
".ipynb": NotebookSplitter(**splitter_kwargs),
+ ".json": RecursiveJsonSplitter(**splitter_kwargs),
}
Optionally, I can help make these changes if you give me permission to push to your fork's branch.
As an immediate workaround for getting context from JSON files, you can use the new context command @file:<file-path>
. When you send a message with this command included, it will include the entire contents of the specified file with your prompt. This is already available in Jupyter AI v2.24.0 and above.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for your contribution! I can help implement my suggestion above in a follow-up PR, no worries.
Oh shoot- I seem to not get emails about this for some reason... Thanks for the follow up + the fix as well! I'll have to figure out how to get notified properly... |
JSON is basically just JS / txt files and I'm kind of getting annoyed that I keep having to rename extension to JS to use (Maybe no limitation on files that aren't binaries?)