Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce ColumnDataType to use as return type to ColumnSource#getType #3455

Open
nbauernfeind opened this issue Feb 21, 2023 · 1 comment
Open

Comments

@nbauernfeind
Copy link
Member

nbauernfeind commented Feb 21, 2023

With the introduction of ZonedDateTime column type return types, it is very easy to lose the ZoneId -- such as via a groupBy and then an ungroup.

Similarly, it might be nice to have a way to maintain the original column type from Arrow sources. If we group/ungroup (and otherwise only affect ordering of rows) it might be expected that Barrage does not change the Arrow source column type. For example, uint4 get upgraded to long internally -- but we may prefer to send uint4 to non-web-client subscribers.

We could introduce a ColumnDataType type to use instead of Class<?> for ColumnSource#getType.

It is desirable to use the QST's Column Data Type as marrying the java client to the server has usability benefits. There is some design to work through w.r.t. Instant as this is how the java client refers to server-side DateTime columns.

@nbauernfeind nbauernfeind added feature request New feature or request query engine labels Feb 21, 2023
@nbauernfeind nbauernfeind changed the title Introduce ColumnType to use as return type to ColumnSource#getType Introduce ColumnDataType to use as return type to ColumnSource#getType Feb 22, 2023
@nbauernfeind nbauernfeind self-assigned this Feb 22, 2023
@rcaudy rcaudy modified the milestones: Backlog, Apr 2023 Feb 27, 2023
nbauernfeind added a commit to deephaven/web-client-ui that referenced this issue Feb 28, 2023
deephaven/deephaven-core#3385 is a port of DH-11692 which adds `Instant`
and `ZonedDateTime` support to the engine. I've similarly encoded these
types over Barrage as a `long` (as a nanosecond since epoch). The
encoding of Zones will actually need to be implemented at a future date
as the best implementation requires (encourages?) a much larger
refactoring of ColumnSource types. See deephaven/deephaven-core#3455 for
more information.

I was able to get the web-client-ui to display these types with this
small patch.
nbauernfeind added a commit that referenced this issue Mar 3, 2023
DH-11692 via commits:
- deephaven-ent/iris@563e69e
- deephaven-ent/iris@f680b26
- deephaven-ent/iris@4f46299
- deephaven-ent/iris@7baba9f

Follow up work:
- Instant/LocalDate/LocalTime/ZonedDateTime support for Parquet, KafkaTools and JsonNodeUtil #3367
- ArraySource#move support moving of full blocks #3359
- Introduce ColumnDataType to use as return type to ColumnSource#getType #3455
@nbauernfeind
Copy link
Member Author

Also see #3650.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants