feat: upgrade to Arrow 37 and Datafusion 23 #1314
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was discussed with @wjones127 in Slack
This has lots of API changes and will need followup work
This feature makes it much easier to read/write JSON to parquet. This is immediately useful in kafka-delta-ingest but I believe will be much more generally useful for all consumers of the package
I figure we can use this feature for other JSONy things too
|
ACTION NEEDED delta-rs follows the Conventional Commits The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification. |
Most of what appears to have changed can be solved by a Vec<ArrowField>.into() to convert to the Fields abstraction
rtyler
changed the title
rtyler
changed the title
This tracks changes in arrow 37 which switched to basically relying on Arc<Field>
wjones127
reviewed
Apr 29, 2023
Comment on lines
-324
to
-334
| pub fn arrow_schema_json(&self) -> PyResult<String> { | ||
| let schema = self | ||
| ._table | ||
| .get_schema() | ||
| .map_err(PyDeltaTableError::from_raw)?; | ||
| serde_json::to_string( | ||
| &<ArrowSchema as TryFrom<&deltalake::Schema>>::try_from(schema) | ||
| .map_err(PyDeltaTableError::from_arrow)?, | ||
| ) | ||
| .map_err(|_| PyDeltaTableError::new_err("Got invalid table schema")) | ||
| } |
There was a problem hiding this comment.
I don't think this was used in our public API, and I know the arrow-rs folks wanted to remove the unofficial JSON schema serialization from their public API as well. I think it should be fine to drop this.
rtyler
requested review from
fvaleye,
roeap,
houqp,
xianwill and
mosyp
as code owners
wjones127
approved these changes
Apr 29, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Lots of fun API changes,
Fieldsis the biggest impact in terms of lines of code however.BREAKING CHANGE: new major versions for arrow and datafusion