rfctr: prep for pluggable partitioners #3806
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
scanny
force-pushed
the
scanny/pluggable-partitioner-prep
branch
2 times, most recently
from
7255b54 to
1584d00
Compare
Coniferish
approved these changes
Dec 9, 2024
scanny
force-pushed
the
scanny/pluggable-partitioner-prep
branch
from
1584d00 to
45e3fc6
Compare
Legacy predecessor to `metadata_filename`. Deprecated for over a year.
Add the flexibility to have multiple return-points within `partition()` by extracting the post-processing performed after partitioning to a function that can be called from whatever exit-points are convenient.
These are going to be harder to unify the signature of, if we ever do, so get these out of the way up front.
Also remove unused public property of `HtmlPartitionerOptions` noticed while we were in there.
Now that the call signature for non-PDF/Image file-types is uniform, dispatch can be reduced to a single call. This sets the stage for calling a partitioner unknown at compile-time, one registered at run-time.
scanny
force-pushed
the
scanny/pluggable-partitioner-prep
branch
from
45e3fc6 to
21342da
Compare
Suspect IssuesThis pull request was deployed and Sentry observed the following issues:
Did you find this useful? React with a 👍 or 👎 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Prepare auto-partitioning for pluggable partitioners.
Move toward a uniform partitioner call signature in
auto/partition()such that a custom or override partitioner can be registered without requiring code changes.Additional Context
The central job of
auto/partition()is to detect the file-type of the given file and use that to dispatch partitioning to the corresponding partitioner function e.g.partition_pdf()orpartition_docx().In the existing code, each partitioner function is called with parameters "hand-picked" from the available parameters passed to the
partition()function. This is unnecessary and couples those partitioners tightly with the dispatch function. The desired state is that all available arguments are passed askwargsand the partitioner function "self-selects" the arguments it will be sensitive to, applies its own appropriate default values when the argument is omitted, and simply ignore any arguments it doesn't use. Note that achieving this requires no changes to partitioner functions because they already do precisely this.So the job is to pass all arguments (other than
filenameandfile) to the partitioner askwargs. This will allow additional or alternate partitioners to be registered at runtime and dispatched to, because as long as they have the signaturepartition_x(filename, file, kwargs) -> list[Element]then they can be dispatched to without customization.