-
Notifications
You must be signed in to change notification settings - Fork 81
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Parquet: Support non-UTC timestamps #976
Comments
Elaborating on the description, the code in |
Parquet handling of isAdjustedToUTC: TIMESTAMP type has two extra fields: isAdjustedToUTC (= true/false), unit (= MILLIS/MICROS/NANOS).
More details in official docs here. |
I think on a column-by-column basis we should be able to add in an assumed timezone; for example, the NYC taxi dataset e, we can assume that the timezone is NYC timezone. As it stands today, we read this in as a LocalDateTime. |
Original issue completed as part of #4801. |
Running (in groovy):
Would result in:
If we didn't have a fortunate schema conversion error that effectively treats the timestamps as UTC. We would like to reverse the error and properly support zoned timestamps. Presumably we could specify a timezone in read instructions.
We do have limited support in:
io.deephaven.db.v2.locations.parquet.topage.ToDBDateTimePageFromInt96
Likely the reference time zone used there should be moved out to
ParquetInstructions
. Then we just need to integrate those instructions into the Int96 support and into the more standard datetime fields.The text was updated successfully, but these errors were encountered: