Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CDC is not generated for Structs and Lists #2568

Closed
rtyler opened this issue Jun 4, 2024 · 0 comments · Fixed by #2727
Closed

CDC is not generated for Structs and Lists #2568

rtyler opened this issue Jun 4, 2024 · 0 comments · Fixed by #2727
Assignees
@rtyler
Labels
binding/rust Issues for the Rust crate bug Something isn't working
Milestone
Change Data Captu...

Comments

@rtyler
Copy link
Member

rtyler commented Jun 4, 2024

This is a placeholder issue for future reference in code warnings.

In short, there is an issue upstream in Datafusion and its Arrow dependency which prevents comparisons happening properly when the Delta table schema contains StructType or ListType, and therefore the CDC code in delta-rs cannot properly compute the before/after difference on a set of data.

Once apache/datafusion#10749 is resolved and released, then we can remove the warning.
@rtyler rtyler added bug Something isn't working binding/rust Issues for the Rust crate labels Jun 4, 2024
@rtyler rtyler self-assigned this Jun 4, 2024
@rtyler rtyler added this to the Change Data Capture Support milestone Jun 4, 2024
rtyler added a commit to rtyler/delta-rs that referenced this issue Jun 4, 2024
@rtyler
chore: provide a warning if the schema contains fields which break CDC
27e3d01 
See delta-io#2568
rtyler added a commit to rtyler/delta-rs that referenced this issue Jun 4, 2024
@rtyler
chore: provide a warning if the schema contains fields which break CDC
9cbd296 
See delta-io#2568
rtyler added a commit to rtyler/delta-rs that referenced this issue Jun 4, 2024
@rtyler
chore: provide a warning if the schema contains fields which break CDC
b154f87 
See delta-io#2568
rtyler added a commit to rtyler/delta-rs that referenced this issue Jun 4, 2024
@rtyler
chore: provide a warning if the schema contains fields which break CDC
d79f7d6 
See delta-io#2568
ion-elgreco pushed a commit to rtyler/delta-rs that referenced this issue Jun 4, 2024
@rtyler @ion-elgreco
chore: provide a warning if the schema contains fields which break CDC
a3b2600 
See delta-io#2568
ion-elgreco pushed a commit that referenced this issue Jun 4, 2024
@rtyler @ion-elgreco
chore: provide a warning if the schema contains fields which break CDC
4a3f74e 
See #2568
@ion-elgreco ion-elgreco mentioned this issue Aug 4, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rtyler rtyler

Labels
binding/rust Issues for the Rust crate bug Something isn't working
Projects
None yet
Milestone
Change Data Capture Support
Development

Successfully merging a pull request may close this issue.

feat: use logical plan in update, refactor/simplify CDCTracker ion-elgreco/delta-rs
1 participant
@rtyler