Some types don't seem to round-trip cleanly through parquet and pandas #3151
Labels
2023_triagedNoMilestone
bug
Something isn't working
parquet
Related to the Parquet integration
python
python-server-side
Milestone
Description
It is possible that this is not a valid bug, but worth investigating. Three expressions in PR #3141 were left commented out because they don't roundtrip cleanly through pandas and parquet, likely due to how python handles these types rather than specifics of parquet or deephaven's internal parquet implementation.
This issue is to follow-up on what is going wrong here, to make sure that these are acceptable, or document how to deal with them.
Basic setup:
"someTime = DateTime.now() + i"
For this, pyarrow emits a message indicating that precision is lost in a way that doesn't make sense to me:
The error acknowledges that the data is formatted as nanosecond, but for some reason is attempting to write to disk as microsecond?
"nullBigDecColumn = (java.math.BigDecimal)null"
and"nullBigIntColumn = (java.math.BigInteger)null"
For both of these, when passed DH -> parquet -> pandas -> DH, the resulting column is a primitive int rather than the java type, likely since there is no data, and with no data the scale ends up as 0.
Versions
The text was updated successfully, but these errors were encountered: