Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support efficient last N rows for stream table #2121

Open
devinrsmith opened this issue Mar 24, 2022 · 2 comments
Open

Support efficient last N rows for stream table #2121

devinrsmith opened this issue Mar 24, 2022 · 2 comments
Assignees
Labels
Milestone

Comments

@devinrsmith
Copy link
Member

We don't currently have efficient support for retaining the last N rows of a stream table.

stream.tail(N) itself is a streaming table (ie, it doesn't retain)

stream.tailBy(N) might have the semantics we are looking for, but the current implementation doesn't work for streaming tables (implemented using groupBy which doesn't work on streaming tables). .tailBy(N) may be more powerful than we need though, since it also supports grouping columns. (Follow-up question if .tailBy(N, ...) should semantically ungroup automatically or not.) That said, retaining the last N rows of a stream table per group may be desired functionality as well.

It should be relatively straightforward in either case; but the case without the grouping columns will probably be easier.

@devinrsmith devinrsmith added this to the April 2022 milestone Mar 24, 2022
@devinrsmith devinrsmith self-assigned this Mar 24, 2022
@devinrsmith
Copy link
Member Author

There is a hacky workaround if you have a stream table with an increasing sequence number (like a kafka table):

last_n = kafka_stream.updateView("X=KafkaOffset % N").lastBy("X")

or

last_n = kafka_stream.updateView("X=KafkaOffset % N").lastBy("X").sort("KafkaOffset")

@devinrsmith
Copy link
Member Author

devinrsmith commented Mar 24, 2022

Assuming a single partition - otherwise, may also want a lastBy("KafkaPartition", "X") (but then, it's not strictly last "N").

@devinrsmith devinrsmith modified the milestones: April 2022, Backlog Mar 25, 2022
mofojed pushed a commit that referenced this issue Jul 17, 2024
## Release notes
https://github.com/deephaven/web-client-ui/releases/tag/v0.86.0

### Features

* Add option to disable WebGL rendering
([#2134](deephaven/web-client-ui#2134))
([011eb33](deephaven/web-client-ui@011eb33))
* Core plugins refactor, XComponent framework
([#2150](deephaven/web-client-ui#2150))
([2571fad](deephaven/web-client-ui@2571fad))
* IrisGridTheme iconSize
([#2123](deephaven/web-client-ui#2123))
([58ee88d](deephaven/web-client-ui@58ee88d)),
closes [#885](deephaven/web-client-ui#885)
* Partitioned Table UI Enhancements
([#2110](deephaven/web-client-ui#2110))
([de5ce40](deephaven/web-client-ui@de5ce40)),
closes [#2079](deephaven/web-client-ui#2079)
[#2066](deephaven/web-client-ui#2066)
[#2103](deephaven/web-client-ui#2103)
[#2104](deephaven/web-client-ui#2104)
[#2105](deephaven/web-client-ui#2105)
[#2106](deephaven/web-client-ui#2106)
[#2107](deephaven/web-client-ui#2107)
[#2108](deephaven/web-client-ui#2108)
[#2109](deephaven/web-client-ui#2109)
[#2049](deephaven/web-client-ui#2049)
[#2120](deephaven/web-client-ui#2120)
[#1904](deephaven/web-client-ui#1904)


### Bug Fixes

* error when edited cell is out of grid viewport
([#2148](deephaven/web-client-ui#2148))
([3fccd43](deephaven/web-client-ui@3fccd43)),
closes [#2087](deephaven/web-client-ui#2087)

## Release notes
https://github.com/deephaven/web-client-ui/releases/tag/v0.85.2


### Bug Fixes

* Fix missing scrim background on LoadingOverlay
([#2098](deephaven/web-client-ui#2098))
([c9ed895](deephaven/web-client-ui@c9ed895))

## Release notes
https://github.com/deephaven/web-client-ui/releases/tag/v0.85.1

##
[0.85.1](deephaven/web-client-ui@v0.85.0...v0.85.1)
(2024-07-08)


### Bug Fixes

* re-export remaining types needed by dh ui from @react-types/shared
([#2132](deephaven/web-client-ui#2132))
([2119a61](deephaven/web-client-ui@2119a61))



## Release notes
https://github.com/deephaven/web-client-ui/releases/tag/v0.85.0

### Features

* ComboBox - @deephaven/jsapi-components
([#2077](deephaven/web-client-ui#2077))
([115e057](deephaven/web-client-ui@115e057)),
closes [#2074](deephaven/web-client-ui#2074)


### Bug Fixes

* Allow ComboBox to accept the FocusableRef for ref
([#2121](deephaven/web-client-ui#2121))
([8fe9bad](deephaven/web-client-ui@8fe9bad))
* Ref was not being passed through for Picker
([#2122](deephaven/web-client-ui#2122))
([a11e2ce](deephaven/web-client-ui@a11e2ce))

Co-authored-by: deephaven-internal <66694643+deephaven-internal@users.noreply.github.com>
mofojed added a commit to mofojed/deephaven-core that referenced this issue Jul 18, 2024
Release notes https://github.com/deephaven/web-client-ui/releases/tag/v0.86.1

* add back panel prop to IrisGrid Plugin ([deephaven#2155](deephaven/web-client-ui#2155)) ([6362eb7](deephaven/web-client-ui@6362eb7))
* Remove the session wrapper from the embedded widget app ([deephaven#2158](deephaven/web-client-ui#2158)) ([b76c1d7](deephaven/web-client-ui@b76c1d7))

* Add option to disable WebGL rendering ([deephaven#2134](deephaven/web-client-ui#2134)) ([011eb33](deephaven/web-client-ui@011eb33))
* Core plugins refactor, XComponent framework ([deephaven#2150](deephaven/web-client-ui#2150)) ([2571fad](deephaven/web-client-ui@2571fad))
* IrisGridTheme iconSize ([deephaven#2123](deephaven/web-client-ui#2123)) ([58ee88d](deephaven/web-client-ui@58ee88d))
* Partitioned Table UI Enhancements ([deephaven#2110](deephaven/web-client-ui#2110)) ([de5ce40](deephaven/web-client-ui@de5ce40))

* error when edited cell is out of grid viewport ([deephaven#2148](deephaven/web-client-ui#2148)) ([3fccd43](deephaven/web-client-ui@3fccd43))

* Fix missing scrim background on LoadingOverlay ([deephaven#2098](deephaven/web-client-ui#2098)) ([c9ed895](deephaven/web-client-ui@c9ed895))

* re-export remaining types needed by dh ui from @react-types/shared ([deephaven#2132](deephaven/web-client-ui#2132)) ([2119a61](deephaven/web-client-ui@2119a61))

* ComboBox - @deephaven/jsapi-components ([deephaven#2077](deephaven/web-client-ui#2077)) ([115e057](deephaven/web-client-ui@115e057))

* Allow ComboBox to accept the FocusableRef for ref ([deephaven#2121](deephaven/web-client-ui#2121)) ([8fe9bad](deephaven/web-client-ui@8fe9bad))
* Ref was not being passed through for Picker ([deephaven#2122](deephaven/web-client-ui#2122)) ([a11e2ce](deephaven/web-client-ui@a11e2ce))

Release notes https://github.com/deephaven/web-client-ui/releases/tag/v0.84.0

* - @deephaven/jsapi-components - The contract of
`useSearchableViewportData` to be more consistent with
`useViewportData`. `usePickerWithSelectedValues` now requires
`timeZone`.
- @deephaven/jsapi-utils - `createSearchTextFilter` requires `timeZone`

* useSearchableViewportData - support non-text filters ([deephaven#2092](deephaven/web-client-ui#2092)) ([7009e21](deephaven/web-client-ui@7009e21))

* `isElementOfType` Improved type inference ([deephaven#2099](deephaven/web-client-ui#2099)) ([e13c9d7](deephaven/web-client-ui@e13c9d7))
* Console does not scroll to bottom when code run from notebook ([deephaven#2114](deephaven/web-client-ui#2114)) ([e75e716](deephaven/web-client-ui@e75e716))
* make textValue default to key for Normalized Item ([deephaven#2113](deephaven/web-client-ui#2113)) ([bd3e944](deephaven/web-client-ui@bd3e944))
* Update IrisGridContextMenuHandler getHeaderActions return type to be more permissive ([deephaven#2117](deephaven/web-client-ui#2117)) ([4e08b79](deephaven/web-client-ui@4e08b79))
devinrsmith pushed a commit that referenced this issue Jul 18, 2024
## Release notes
https://github.com/deephaven/web-client-ui/releases/tag/v0.86.1

### Bug Fixes

* add back panel prop to IrisGrid Plugin
([#2155](deephaven/web-client-ui#2155))
([6362eb7](deephaven/web-client-ui@6362eb7))
* Remove the session wrapper from the embedded widget app
([#2158](deephaven/web-client-ui#2158))
([b76c1d7](deephaven/web-client-ui@b76c1d7))

## Release notes
https://github.com/deephaven/web-client-ui/releases/tag/v0.86.0

### Features

* Add option to disable WebGL rendering
([#2134](deephaven/web-client-ui#2134))
([011eb33](deephaven/web-client-ui@011eb33))
* Core plugins refactor, XComponent framework
([#2150](deephaven/web-client-ui#2150))
([2571fad](deephaven/web-client-ui@2571fad))
* IrisGridTheme iconSize
([#2123](deephaven/web-client-ui#2123))
([58ee88d](deephaven/web-client-ui@58ee88d))
* Partitioned Table UI Enhancements
([#2110](deephaven/web-client-ui#2110))
([de5ce40](deephaven/web-client-ui@de5ce40))


### Bug Fixes

* error when edited cell is out of grid viewport
([#2148](deephaven/web-client-ui#2148))
([3fccd43](deephaven/web-client-ui@3fccd43))

## Release notes
https://github.com/deephaven/web-client-ui/releases/tag/v0.85.2


### Bug Fixes

* Fix missing scrim background on LoadingOverlay
([#2098](deephaven/web-client-ui#2098))
([c9ed895](deephaven/web-client-ui@c9ed895))

## Release notes
https://github.com/deephaven/web-client-ui/releases/tag/v0.85.1

##
[0.85.1](deephaven/web-client-ui@v0.85.0...v0.85.1)
(2024-07-08)


### Bug Fixes

* re-export remaining types needed by dh ui from @react-types/shared
([#2132](deephaven/web-client-ui#2132))
([2119a61](deephaven/web-client-ui@2119a61))



## Release notes
https://github.com/deephaven/web-client-ui/releases/tag/v0.85.0

### Features

* ComboBox - @deephaven/jsapi-components
([#2077](deephaven/web-client-ui#2077))
([115e057](deephaven/web-client-ui@115e057))


### Bug Fixes

* Allow ComboBox to accept the FocusableRef for ref
([#2121](deephaven/web-client-ui#2121))
([8fe9bad](deephaven/web-client-ui@8fe9bad))
* Ref was not being passed through for Picker
([#2122](deephaven/web-client-ui#2122))
([a11e2ce](deephaven/web-client-ui@a11e2ce))


## Release notes
https://github.com/deephaven/web-client-ui/releases/tag/v0.84.0

### ⚠ BREAKING CHANGES

* - @deephaven/jsapi-components - The contract of
`useSearchableViewportData` to be more consistent with
`useViewportData`. `usePickerWithSelectedValues` now requires
`timeZone`.
- @deephaven/jsapi-utils - `createSearchTextFilter` requires `timeZone`

### Features

* useSearchableViewportData - support non-text filters
([#2092](deephaven/web-client-ui#2092))
([7009e21](deephaven/web-client-ui@7009e21))


### Bug Fixes

* `isElementOfType` Improved type inference
([#2099](deephaven/web-client-ui#2099))
([e13c9d7](deephaven/web-client-ui@e13c9d7))
* Console does not scroll to bottom when code run from notebook
([#2114](deephaven/web-client-ui#2114))
([e75e716](deephaven/web-client-ui@e75e716))
* make textValue default to key for Normalized Item
([#2113](deephaven/web-client-ui#2113))
([bd3e944](deephaven/web-client-ui@bd3e944))
* Update IrisGridContextMenuHandler getHeaderActions return type to be
more permissive
([#2117](deephaven/web-client-ui#2117))
([4e08b79](deephaven/web-client-ui@4e08b79))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant