Support configurable output timezone in pre_transform_spec #116
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… custom format timezone
When true, datasets with inline values are extracted onto the server spec. When false, these datasets are left in the client spec. When performing pre_transform_spec on the server, it's preferable to set this to true. But if planning is done on the client, then the past default of false is more appropriate.
This format is found throughout the vega-lite examples
Vega skips over the bad values since each row can have a different type. When arrow reads this the values column is inferred as String type, and all of the numeric values are read as null.
Explanation: The way arrow parses this is for the datum.options.price to be null, but not datum.options.
Vega fails when this is present in the pre-transformed version of the spec.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds a new argument to
pre_transform_speccalledoutput_tz. This argument overrides the timezone that local datetime values are converted to before being stored in the pre-transformed spec as naive date strings. When not provided,output_tzdefaults tolocal_tz, which matches the existing behavior.Along the way, i needed to update the DataFusion revision to capture the fix in apache/datafusion#2463.
I also ran into windows CI issues that seem to have boiled down to the builds running out of space. What finally solved it was moving the build target directories to the large
C:\drive.