PyPDFLoader use url in metadata source if file is a web path

[123-fake-st]

Description: Update langchain.document_loaders.pdf.PyPDFLoader to store url in metadata (instead of a temporary file path) if user provides a web path to a pdf

• Issue: Related to Loading online PDFs gives temporary file path as source in metadata #7034; the reporter on that issue submitted a PR updating PyMuPDFParser for this behavior, but it has unresolved merge issues as of 20 Oct 2023 Changed the metadata source from temporary file location to original … #7077

  • In addition to PyPDFLoader and PyMuPDFParser, these other classes in langchain.document_loaders.pdf exhibit similar behavior and could benefit from an update: PyPDFium2Loader, PDFMinerLoader, PDFMinerPDFasHTMLLoader, PDFPlumberLoader (I'm happy to contribute to some/all of that, including assisting with PyMuPDFParser, if my work is agreeable)
  • The root cause is that the underlying pdf parser classes, e.g. langchain.document_loaders.parsers.pdf.PyPDFParser, never receive information about the url; the parsers receive a langchain.document_loaders.blob_loaders.blob, which contains the pdf contents and local file path, but not the url
  • This update passes the web path directly to the parser since it's minimally invasive and doesn't require further changes to maintain existing behavior for local files... bigger picture, I'd consider extending blob so that extra information like this can be communicated, but that has much bigger implications on the codebase which I think warrants maintainer input

• Dependencies: None

# old behavior
>>> from langchain.document_loaders import PyPDFLoader
>>> loader = PyPDFLoader('https://arxiv.org/pdf/1706.03762.pdf')
>>> docs = loader.load()
>>> docs[0].metadata
{'source': '/var/folders/w2/zx77z1cs01s1thx5dhshkd58h3jtrv/T/tmpfgrorsi5/tmp.pdf', 'page': 0}

# new behavior
>>> from langchain.document_loaders import PyPDFLoader
>>> loader = PyPDFLoader('https://arxiv.org/pdf/1706.03762.pdf')
>>> docs = loader.load()
>>> docs[0].metadata
{'source': 'https://arxiv.org/pdf/1706.03762.pdf', 'page': 0}

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
langchain	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Oct 27, 2023 5:08pm

[eyurtsev]

hi @123-fake-st thanks for the contribution! I think this has to be fixed at the blob level. I would rather avoid adding complexity to the parsers while they're simple since it's hard to deprecate it afterwards as users start to depend on these parameters.

[123-fake-st]

hi @eyurtsev: thank you for the review and of course to you and the team in general for the work on this project. Your response makes sense and I had been stuck on the idea that blob and/or Blob.from_path() would need to be modified to allow web path information to be passed in. This update reverts changes to the pdf parser and limits code changes to PyPDFLoader only, while retaining the desired new behavior.

[123-fake-st]

Reverted all changes to langchain.document_loaders.parsers.pdf.PyPDFParser and updated langchain.document_loaders.pdf.PyPDFLoader to be able to create a langchain.document_loaders.blob_loaders.blob that has pdf data and uses a web url as the path, when appropriate.

[eyurtsev]

This works!