Shreyas/unstructured

[shreyaspimpalgaonkar]

🚀	This description was created by Ellipsis for commit c776bf1

Summary:

This PR updates dependencies and integrates a new unstructured parsing and chunking strategy, modifying configurations, providers, and services.

Key points:

• Update python-pptx to ^1.0.1 in pyproject.toml.
• Add unstructured dependency with all-docs extras.
• Modify chunk method in py/core/base/providers/chunking.py to accept Union[str, DocumentExtraction].
• Update ParsingConfig to include chunking_config.
• Add unstructured.toml for parsing and chunking configurations.
• Pass chunking_config in R2RConfig.
• Handle unstructured_api in create_parsing_provider.
• Enhance error logging in py/core/pipes/ingestion/chunking_pipe.py.
• Log errors and handle DocumentExtraction in ChunkingPipe.
• Handle DocumentExtraction in R2RChunkingProvider.
• Yield text directly in UnstructuredChunkingProvider if partitioned.
• Support API-based parsing in UnstructuredParsingProvider.

---

Generated with ❤️ by ellipsis.dev

[ellipsis-dev]

❌ Changes requested. Reviewed everything up to f8aa82a in 35 seconds

More details

• Looked at 368 lines of code in 11 files
• Skipped 0 files when reviewing.
• Skipped posting 3 drafted comments based on config settings.

1. r2r/providers/chunking/unstructured_chunking.py:29

• Draft comment:
  Ensure that parsed_document.data is already chunked or processed appropriately before yielding it directly when partitioned_by_unstructured is True. This could lead to incorrect processing if the data is not in the expected format.
• Reason this comment was not posted:
  Decided after close inspection that this draft comment was likely wrong and/or not actionable:
  The comment is speculative and asks for verification, which violates the rules. It does not point out a definite issue but rather suggests a potential one. The code seems to handle the condition where partitioned_by_unstructured is True by directly yielding the data, which is likely intentional.
  I might be overlooking a potential issue with the data format when partitioned_by_unstructured is True, but the comment does not provide a definite problem, only a speculative one.
  Even if there is a potential issue, the comment does not provide a clear, actionable change, and thus does not meet the criteria for a useful comment.
  Remove the comment as it is speculative and does not point out a definite issue, violating the rules.

2. r2r/providers/parsing/unstructured_parsing.py:88

• Draft comment:
  Ensure that element is always a dictionary before iterating over it. Add a check or handle cases where element might not be a dictionary to avoid runtime errors.
• Reason this comment was not posted:
  Decided after close inspection that this draft comment was likely wrong and/or not actionable:
  The code already includes a check to ensure 'element' is a dictionary before iterating over it. The comment seems redundant because the check is already implemented. The comment does not suggest any additional code change that is required.
  I might be overlooking a scenario where 'element' could be something other than a dictionary, but the existing check seems to handle this.
  The existing check for 'element' being a dictionary should suffice, and the comment does not add any new information or suggest a necessary change.
  The comment is unnecessary because the code already checks if 'element' is a dictionary. It should be removed.

3. r2r/main/assembly/factory.py:97

• Draft comment:
  Consider explicitly passing a boolean value for use_api when initializing UnstructuredParsingProvider for better clarity and to avoid potential logical errors.
• Reason this comment was not posted:
  Confidence changes required: 50%
  In r2r/main/assembly/factory.py, the create_parsing_provider method has a potential issue with the way it initializes UnstructuredParsingProvider. The boolean logic for use_api is directly passed, which might not be clear. It would be better to explicitly pass a boolean value for clarity.

Workflow ID: wflow_q4rCzkTx0UksAh9x

---

Want Ellipsis to fix these issues? Tag @ellipsis-dev in a comment. You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.

[ellipsis-dev]

👍 Looks good to me! Incremental review on a702131 in 18 seconds

More details

• Looked at 27 lines of code in 2 files
• Skipped 0 files when reviewing.
• Skipped posting 1 drafted comments based on config settings.

1. py/core/main/services/ingestion_service.py:385

• Draft comment:
  The exc_info=True parameter was removed from the logger.error call, which may reduce the amount of information logged during an error. Consider whether this change is intentional and if the stack trace is needed for debugging.
• Reason this comment was not posted:
  Confidence changes required: 50%
  The PR description mentions removing exc_info=True from the logger.error call, but this change is not reflected in the code. This discrepancy should be addressed.

Workflow ID: wflow_a3Zb0ogzO0A3Fnki

---

You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.

[ellipsis-dev]

👍 Looks good to me! Incremental review on c776bf1 in 25 seconds

More details

• Looked at 92 lines of code in 4 files
• Skipped 0 files when reviewing.
• Skipped posting 2 drafted comments based on config settings.

1. py/core/providers/chunking/r2r_chunking.py:11

• Draft comment:
  Duplicate import of ChunkingProvider and Method. Remove this line to clean up the code.
• Reason this comment was not posted:
  Confidence changes required: 10%
  The import statement for ChunkingProvider and Method is duplicated, which is unnecessary and can be removed to clean up the code.

2. py/core/providers/chunking/unstructured_chunking.py:26

• Draft comment:
  Ensure parsed_document is either a string or DocumentExtraction to prevent runtime errors. Consider adding a type check and raise an appropriate error if the type is unexpected.
• Reason this comment was not posted:
  Confidence changes required: 30%
  The chunk method in UnstructuredChunkingProvider should handle cases where parsed_document is not a string or DocumentExtraction. This could lead to runtime errors if unexpected types are passed.

Workflow ID: wflow_grAcjbX7Sep07FsN

---

You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.