-
Notifications
You must be signed in to change notification settings - Fork 2.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Core: Ignore split offsets array when split offset is past file length #8925
Core: Ignore split offsets array when split offset is past file length #8925
Conversation
@amogh-jahagirdar, maybe this time we should create a test to validate |
114387d
to
0ff2819
Compare
Done, added a new test which will exercise both |
0ff2819
to
34eee1b
Compare
34eee1b
to
7246d78
Compare
Follow up from a miss in the fix done in #8860. There's an additional splitOffsetArray method which gets used here
iceberg/core/src/main/java/org/apache/iceberg/BaseContentScanTask.java
Line 134 in aa891ac
Alternatively, we can undo the optimization done here: #8336 (comment) and then the original fix in #8860 would be sufficient, and be simpler. Since the optimization seemed impactful and there was community interest in it, this fix retains it and currently just fixes both cases explicitly.