Add full Google Drive features.

[pprados]

Reimplement the Google Drive features

Propose :

• langchain.docstore.GoogleDriveDocStore
• langchain.document_loaders.GoogleDriveLoader
• langchain.utilities.GoogleDriveAPIWrapper
• langchain.tools.GoogleDriveSearchTool
• langchain.utilities.GoogleDriveUtilities

Features:

• Fully compatible with Google Drive API
• Manage file in trash
• Manage shortcut
• Manage file description
• Paging with request GDrive list()
• Multiple kind of template for request GDrive
• Convert a lot of mime type (can be configured). The list is adjusted according to the packages availables
• Convert GDoc, GSheet and GSlide with differents modes
• Can use only the description of files, without loading and conversion of the body
• Lambda fine filter
• Remove duplicate documents (in case of shortcut)
• Add Url to documents (or part of documents like specific slide)
• Use environment variable for reference an API tokens
• Manage different king of strange state with Google File (absence of URL, etc.)
• Use fully lazy strategy to save memory

Recognition

If you accept my pull-request, you can mention me @pprados. Thanks

Before submitting

Unit-tests coverage >80% of new code

No integration test, but some notebook to show how to use.

• docs/modules/agents/tools/examples/google_drive.ipynb
• docs/modules/indexes/document_loaders/examples/google_drive.ipynb
• docs/modules/indexes/retrievers/examples/google_drive.ipynb

Who can review?

Community members can review the PR once tests pass. Tag maintainers/contributors who might be interested:

@eyurtsev @hwchase17 @vowelparrot might be interested

[pprados]

@vowelparrot I have 3 workflow awaitings approval. You must accept to start these jobs?

[Yvelo]

I'd gladly collaborate on improving Google Drive support on LangChain. The above test failled since googleapiclient is a new requirement for poetry.lock.

[pprados]

I resolve this problem. I test the github action in my branch, and now all workflow are correct.

[pprados]

Sorry. I fix the format now. It's not possible to start by hand, the workflow lint

[pprados]

@eyurtsev Can you run the workflows to validate this version ?

[pprados]

After this pull-request, I will normalize the link with google token, to use one scope for google drive, gmail, etc.

[pprados]

@eyurtsev, can you active the workflows, and if all is correct, can you review this code?

[pprados]

@eyurtsev, sorry for the last error.
In my branch, the workflows run without error.

[pprados]

@eyurtsev, I have a question about my implementation.

In the method _lazy_load_file_from_file(), I had an optional run_manager.
I can use this manager if I catch an exception.

if isinstance(run_manager, CallbackManagerForToolRun):
  run_manager.on_tool_error(e)
elif isinstance(run_manager, CallbackManagerForChainRun):
  run_manager.on_chain_error(e)

But the code is not clean. Do you have a better idea?

I initiate the possibility to use a lazy approach in the retriever.

def lazy_get_relevant_documents(self,query:str) -> Iterator[Document]:
   ...

The default implementation transforms a classic list of documents to an iterator. But, the subclasses can be choice to implements a lazy approach, to optimize the memory footprint.

In my Google Drive utilities, I use a lazy approach.

Later, I would like to update the link between the loaders and vectordb, to use a lazy approach if it's possible. Then, a loader can return a big number of documents to import, without problems with the memory.

[pprados]

The code changes every day, so I must make rebase another time. @eyurtsev, If you star the workflows quickly, all will be correct ;-)

[pprados]

Hello @hwchase17,
The master moves every day. So, sometime I have the 3 workflows OK, sometime I have no conflicts with the base branch.
Else, I must rebase every day.
Can you analyze this pull request or find someone to do it?

Thanks

[pprados]

@hwchase17, @eyurtsev,
Another time, all checks have passed, but I have a conflicting files: poetry.lock.

I rebase my code with the last version.
Can you review the code before the next release ?
Please ;-)

[pprados]

Hello @eyurtsev, @hwchase17,

I'm sure you've got plenty of pull requests to validate.
New version with a rebase from 12 June.
Maybe, other people can revu my code.
I have been waiting for 3 weeks now.
Thanks

[pprados]

Hello @eyurtsev, @hwchase17, @vowelparrot, @baskaryan and @hinthornw,

Can someone contact me via discord, to organize a commented review, if that would make things easier?

[pprados]

#9999