[BUG] Chunked ORC writer throws in close() if not tables have been written successfully
#15386
Comments
3 tasks
rapids-bot bot
pushed a commit
that referenced
this issue
Apr 2, 2024
…ritten (#15393) Closes #15386, #15387 The fixes for the two issues overlap, so I included both in a single PR. Expanded the `_closed` flag to an enum that tracks if the operations in `close()` should be performed (one or more tables were written to the sink). This way, we don't perform the steps in close when there is no valid file to write the footer for. This includes: - No `write` calls; - All `write` calls failed; The new enum replaces `skip_close()` that used to fix this issue for a smaller subset of cases. Additionally, writing of the ORC header has been moved after the encode and uses the new state to only write the header in the first `write` call. This way we don't write anything to the sink if there were no `write` calls with the writer, and if the encode failed in the `write`s. Authors: - Vukasin Milovanovic (https://github.com/vuule) - Nghia Truong (https://github.com/ttnghia) Approvers: - Nghia Truong (https://github.com/ttnghia) - David Wendt (https://github.com/davidwendt) URL: #15393
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
finish_statistic_blobsinclosethrows because the footer is empty, and several vector sizes are based onfooter.types.size() - 1.Proposal:
We should not perform the actions in
closeif no data was written successfully. Ideally we would not write anything to the sink in such case.The text was updated successfully, but these errors were encountered: