[Logs UI] [R&D] Investigate EuiDataGrid for log stream

[weltenwort]

Summary

The log stream has been primarily implemented to support infinite scrolling in both the up and down direction. The introduction of the date range selector a few months ago changed some assumptions that guided the initial implementation. In the mean time the <EuiDataGrid> was added to EUI and is used in the "document explorer" mode of Kibana Discover. If the log stream was based on <EuiDataGrid> it could share more code with Discover or even become just a special viewing mode in Discover. The the same time it would benefit from the various new features and usability improvements added to Discover over the years.

Task breakdown

• Set up experimentation environment in the data grid docs in the EUI repo
• Prove that bidirectional scrolling is possible using the react-window API
• Determine whether <EuiDataGrid> can offer an abstraction over the infinite scrolling / incremental loading.

Questions

⚠️ This is not an exhaustive list. More questions can and hopefully will arise during the investigation.

• Can more of the react-window API be exposed on the <EuiDataGrid> to allow for sufficient control over the pagination to emulate a virtualized scrolling until [EuiDataGrid] bi-directional infinite scrolling eui#2981 is solved?
• Or would it be feasible/efficient to solve [EuiDataGrid] bi-directional infinite scrolling eui#2981 directly by adding infinite scrolling as first-class pagination mode?

Related links

• EuiDataGrid documentation: https://elastic.github.io/eui/#/tabular-content/data-grid
• virtualized scrolling issue: [EuiDataGrid] bi-directional infinite scrolling eui#2981
• WIP of the Discover team: [Discover] EuiDataGrid Implementation #67259
• analysis of the Lens team: [Lens] Replace basic table with EuiDataGrid #72504

[elasticmachine]

Pinging @elastic/logs-metrics-ui (Team:logs-metrics-ui)

[afgomez]

I made a list of features that we have right now in the log stream, and if the EuiDataGrid satisfies them or not.

The datagrid has scroll	✅	The body of the datagrid can be made scrollable with a style={{ height: "1234px" }} prop
The scroll of the body is independent of the header	✅
The component reports scroll position	✅	The scroll position can be read through the `onScrollCapture` prop.
Elements can be added dynamically	⚠️	The scroll jumps when rows are prepended. This will need to be addressed somehow.
Can render special rows (i.e. timestamp markers, top and bottom spinners when loading an extra page, etc).	❌	It doesn't seem possible
Complex cell rendering, for highlighted words	✅
Highlighting of specific rows (besides hovering)	❓	Might be possible with the complex cell rendering. I need to dig more.
The component reports the scroll position of their children. (Needed to jump to specific time cursors)	❌	It doesn't seem possible

[jasonrhodes]

@afgomez @weltenwort do you both think the "X" values in this R&D table are deal-breakers? Should we:

a) Investigate further and work with EUI to think about addressing the short-comings, or
b) Retire this idea for now

Thanks!

[weltenwort]

My gut feeling is that we just need a few more hooks into the rendering process (such as the ability to provider a renderRow override or pass a ref to the table body to listen to scroll events) in order to implement our desired behavior. I wonder whether the EUI team would be opposed to cluttering the component's API in that way.

To determine whether it's worth it we should probably try to predict whether we'll soon add anything to the log stream that would require significant changes anyway. In that case it might be worth investing in the EuiDataGrid.

[weltenwort]

ℹ️ We're hoping to start on this very soon. To that end I've updated the description with a task breakdown.

[weltenwort]

ℹ️ Progress update

In the branch https://github.com/weltenwort/eui/tree/data-grid-scrolling-experiment we set up a sandbox to recreate the scenario of incrementally loading log entries when scrolling into either direction. It can be found in the documentation at "Tabular content -> Data grid advanced -> Scrolling and incremental loading". It employs a combination of these mechanisms:

• It implements layout shift compensation when auto-sizing row in the top overscan area. This is tracked in [EuiDataGrid] Reduce layout shift when row heights are measured automatically eui#6024 and will be solved in a corresponding enhancement PR.
• It simulates the loading of chunks of consecutive log entries.
• It reserves a large number of virtual rows (10000) but limits the scrolling to the area backed by actually loaded or loading rows.
• It triggers asynchronous prepending/appending of new chunks when the user scrolls to the boundaries of the loaded chunks.

We are working on two follow-up tasks:

• [DataGrid] Ensure referential integrity of focused cell by Kerry350 · Pull Request #6007 · elastic/eui
• [EuiDataGrid] Reduce layout shift when row heights are measured automatically · Issue #6024 · elastic/eui

[weltenwort]

🏁 This specific investigation and follow-up PRs to EUI have been completed.