Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Json error: Binary is not supported by JSON when writing checkpoint files #1493

Closed
ChewingGlass opened this issue Jun 26, 2023 · 0 comments · Fixed by #1498
Closed

Json error: Binary is not supported by JSON when writing checkpoint files #1493

ChewingGlass opened this issue Jun 26, 2023 · 0 comments · Fixed by #1498
Labels
bug Something isn't working

Comments

@ChewingGlass
Copy link
Contributor

ChewingGlass commented Jun 26, 2023

Environment

Delta-rs version: Latest (master)

Environment:

  • Cloud provider: AWS
  • OS: OSX, docker debian
  • Other:

Bug

What happened:

When writing checkpoints for a table with binary columns via:

        checkpoints::create_checkpoint(&table).await?;

I get the error

Error: Failed to convert into Arrow schema: Json error: Binary is not supported by JSON

Caused by:
    Json error: Binary is not supported by JSON

What you expected to happen:

The checkpoint file should be written

How to reproduce it:

Write a table with binary columns and try to write a checkpoint

More details:

Here's the relevant logs from before the error:

[2023-06-26T14:00:57Z DEBUG deltalake::checkpoints] Writing parquet bytes to checkpoint buffer.
[2023-06-26T14:00:57Z DEBUG deltalake::checkpoints] Writing to checkpoint parquet buffer...
@ChewingGlass ChewingGlass added the bug Something isn't working label Jun 26, 2023
ChewingGlass added a commit to helium/delta-rs that referenced this issue Jun 28, 2023
ChewingGlass added a commit to helium/delta-rs that referenced this issue Jun 28, 2023
ChewingGlass added a commit to helium/delta-rs that referenced this issue Jun 28, 2023
ChewingGlass added a commit to helium/delta-rs that referenced this issue Jun 28, 2023
ChewingGlass added a commit to helium/delta-rs that referenced this issue Jun 28, 2023
ChewingGlass added a commit to helium/delta-rs that referenced this issue Jun 28, 2023
wjones127 pushed a commit to helium/delta-rs that referenced this issue Jun 28, 2023
rtyler added a commit that referenced this issue Jul 15, 2023
…1498)

# Description
Avoid writing statistics for binary columns to fix JSON error thrown by
Arrow

# Related Issue(s)
 - closes #1493

---------

Co-authored-by: Will Jones <willjones127@gmail.com>
Co-authored-by: R. Tyler Croy <rtyler@brokenco.de>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant