Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Multi-Partition parquet:kv tables do not display multiple partition drop-downs #1438

Closed
robbcamera opened this issue Aug 4, 2023 · 4 comments · Fixed by #1580
Closed

Multi-Partition parquet:kv tables do not display multiple partition drop-downs #1438

robbcamera opened this issue Aug 4, 2023 · 4 comments · Fixed by #1580
Assignees
@georgecwan
Labels
bug Something isn't working

Comments

@robbcamera
Copy link

Description

Multi-Partition parquet:kv tables do not display multiple partition drop-downs

Steps to reproduce

  1. place a .parquet file in a multi-partition kv directory. for example, "/path/to/somewhere/date_partition=2023-08-01/id_partition=id-001/my.parquet"
  2. emulate multiple partitions via soft-link; from "/path/to/somewhere" directory, ln -s date_partition=2023-08-01/id_partition=id-001 date_partition=2023-08-01/id_partition=id-002 and ln -s date_partition=2023-08-01 date_partition=2023-08-02. this will make it seem that there are 2 "date" partitions, each of which has 2 "id" partiions
  3. import the table(s) in a query; `ParquetTools.readTable("/path/to/somewhere")
  4. the "most recent" date-partion (2023-08-02) will be selected from the drop-down of available "date" partitions. note that there no is drop-down for the "id" partition

Expected results

  1. a "date" partition and an "id" partition are visible as drop-downs. i guess this gets nasty if i want to select the "date=2023-08-01" and "ALL" of the "id" partitions ...

Actual results

  1. there is no way to select a specific "id"

Additional details and attachments

Versions

  • Engine Version: ...
  • Web UI Version: ...
  • Java Version: ...
  • Barrage Version: ...
  • OS: ...
  • Browser: ...
  • Docker: ...
@robbcamera robbcamera added the bug Something isn't working label Aug 4, 2023
@mofojed mofojed changed the title Multi-Partitiondo not display multiple partition drop-downs Multi-Partition parquet:kv tables do not display multiple partition drop-downs Aug 4, 2023
@mofojed
Copy link
Member

mofojed commented Aug 4, 2023

Web UI only reads one partition column:

web-client-ui/packages/iris-grid/src/IrisGrid.tsx

Line 841 in e3db6bf

const column =

I think it's all wired up through the JS API supporting multiple partition columns, so it's just the UI that needs to update. Parquet is the easiest way to reproduce.

@mofojed mofojed mentioned this issue Aug 21, 2023
Closed
@mofojed
Copy link
Member

mofojed commented Sep 1, 2023

You can make a table with multiple partitions fairly easy in deephaven-core:

import deephaven.plot.express as dx
stocks = dx.data.stocks()
stocks_partitioned = stocks.partition_by(['sym', 'exchange', 'side'])

@mofojed
Copy link
Member

mofojed commented Sep 14, 2023

Will need some guidance from @dsmmcken on layout for multiple partition selectors.

@mofojed
Copy link
Member

mofojed commented Oct 10, 2023 •
edited
Loading

An example of a multiple partition table that doesn't require #1143 :

from deephaven import empty_table

part = empty_table(4).update("II=ii")

from deephaven.parquet import write, read

write(part, "/tmp/pt-1438/x=0/y=0/part.parquet")
write(part, "/tmp/pt-1438/x=0/y=1/part.parquet")
write(part, "/tmp/pt-1438/x=1/y=0/part.parquet")
write(part, "/tmp/pt-1438/x=1/y=1/part.parquet")

partition_table = read("/tmp/pt-1438")

@georgecwan georgecwan mentioned this issue Oct 18, 2023
Merged
georgecwan added a commit that referenced this issue Nov 2, 2023
@georgecwan
feat: Add support for multi-partition parquet:kv tables (#1580)
d92c91e 
Added feature to `IrisGridPartitionSelector` to display additional
partition selector dropdowns when more than one partition column is
present. The values displayed in the dropdown depend on the selected
partitions of columns to the left of it.

The "Append Command" button and associated functions are removed in
preparation for #1143.

Closes #1438

#### Testing instructions:
1. Create and open a multi-partition table:
```python
from deephaven import empty_table

part = empty_table(4).update("II=ii")

from deephaven.parquet import write, read

write(part, "/tmp/pt-1438/x=0/y=0/part.parquet")
write(part, "/tmp/pt-1438/x=0/y=1/part.parquet")
write(part, "/tmp/pt-1438/x=1/y=0/part.parquet")
write(part, "/tmp/pt-1438/x=1/y=1/part.parquet")

partition_table = read("/tmp/pt-1438")
```
2. Use the dropdown to select any combination of partitions and check
that it updates the table correctly.
* The dropdown should only display values that are available after
filtering for all the partitions left of it. (E.g. A state/province
dropdown should only contain the states/provinces corresponding to the
selected country.)
3. Test filtering by typing directly in the dropdown and searching by
using the search bar above the list of available partitions.
4. Retest using other data types for partition columns (dates, strings,
chars, doubles). E.g:
```python
from deephaven.parquet import write, read

write(part, "/tmp/pt2-1438/date_partition=2023-08-01/id_partition=id-001/decimal=0.1/part.parquet")
write(part, "/tmp/pt2-1438/date_partition=2023-08-01/id_partition=id-002/decimal=0.15/part.parquet")
write(part, "/tmp/pt2-1438/date_partition=2023-08-02/id_partition=id-001/decimal=0.08/part.parquet")
write(part, "/tmp/pt2-1438/date_partition=2023-08-02/id_partition=id-002/decimal=0.08/part.parquet")
write(part, "/tmp/pt2-1438/date_partition=2023-08-02/id_partition=id-003/decimal=0.1/part.parquet")

partition_table = read("/tmp/pt2-1438")
```

---------

Co-authored-by: georgecwan <georgecwan@users.noreply.github.com>
@mofojed mofojed mentioned this issue Nov 3, 2023
Merged
mofojed pushed a commit to deephaven/deephaven-core that referenced this issue Nov 3, 2023
@github-actions @deephaven-internal
Update web version 0.53.0 (#4770)
77b25b0 
Release notes https://github.com/deephaven/web-client-ui/releases/tag/v0.53.0

# [0.53.0](deephaven/web-client-ui@v0.52.0...v0.53.0) (2023-11-03)


### Bug Fixes

* DH-15864: Scroll position StuckToBottom shouldn't trigger sharing dot ([#1617](deephaven/web-client-ui#1617)) ([3d4499b](deephaven/web-client-ui@3d4499b))
* Panel focus throwing an exception ([#1609](deephaven/web-client-ui#1609)) ([9e8b7ae](deephaven/web-client-ui@9e8b7ae))
* Plugins were re-registering on every re-render ([#1613](deephaven/web-client-ui#1613)) ([5977389](deephaven/web-client-ui@5977389))


### Features

* Add support for multi-partition parquet:kv tables ([#1580](deephaven/web-client-ui#1580)) ([d92c91e](deephaven/web-client-ui@d92c91e)), closes [#1143](deephaven/web-client-ui#1143) [#1438](deephaven/web-client-ui#1438)
* Babel Plugin - Mock css imports ([#1607](deephaven/web-client-ui#1607)) ([787c542](deephaven/web-client-ui@787c542)), closes [#1606](deephaven/web-client-ui#1606)
* Convert DashboardPlugins to WidgetPlugins ([#1598](deephaven/web-client-ui#1598)) ([a260842](deephaven/web-client-ui@a260842)), closes [#1573](deephaven/web-client-ui#1573)


### Reverts

* "fix: stuck to bottom on filter clear" ([#1616](deephaven/web-client-ui#1616)) ([806a6b6](deephaven/web-client-ui@806a6b6)), closes [deephaven/web-client-ui#1579](deephaven/web-client-ui#1579) [#1615](deephaven/web-client-ui#1615)

Co-authored-by: deephaven-internal <deephaven-internal@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@georgecwan georgecwan

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat: Add support for multi-partition parquet:kv tables georgecwan/web-client-ui
3 participants
@mofojed @georgecwan @robbcamera