[Discover] Allow to load documents beyond the limit of discover:sampleSize #155019
Assignees
Labels
enhancement
New value added to drive a business result
Feature:Discover
Discover Application
impact:high
Addressing this issue will have a high level of impact on the quality/strength of our product.
loe:medium
Medium Level of Effort
Team:DataDiscovery
Discover, search (e.g. data plugin and KQL), data views, saved searches. For ES|QL, use Team:ES|QL.
Comments
kertal
added
Feature:Discover
|
Pinging @elastic/kibana-data-discovery (Team:DataDiscovery) |
kertal
changed the title
kertal
added
impact:low
kertal
added
impact:high
jughosta
added a commit
that referenced
this issue
Aug 17, 2023
…63784) > [!WARNING] > Sorry, I had to recreate the PR #157241 > Please submit your review again. - Closes #155019 Per docs https://www.elastic.co/guide/en/elasticsearch/reference/current/paginate-search-results.html <img width="851" alt="Screenshot 2023-05-10 at 10 25 20" src="https://github.com/elastic/kibana/assets/1415710/b4b9fef4-7dd8-40ed-8244-343889fc4367"> ## Summary 1. This PR improves `search_after` pagination for `date_nanos` time fields. `sort` value will be returned from ES as a string instead of a rounded and incorrect timestamp. This change allows to also simplify logic on Surrounding document page. Before: <img width="400" alt="Screenshot 2023-05-08 at 17 36 19" src="https://github.com/elastic/kibana/assets/1415710/fd9f45c4-5dc2-4103-83b9-8810e3a6e0df"> After: <img width="400" alt="Screenshot 2023-05-08 at 17 37 13" src="https://github.com/elastic/kibana/assets/1415710/fe9090c0-2116-4f77-9a57-a96ae6b00365"> 2. Also in this PR we now allow users to load more documents within the same time range. Once the button is pressed, it will load next portion of documents (same "sampleSize" value will be used). Currently, we limit max total loaded documents to 10000. "Load more" demo:  If refresh interval is on, the button becomes disabled:  Date nanos demo:  100x Flaky test runner https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2801 --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Discover currently loads a maximum of documents defined by
discover:sampleSizein Advanced settings. There's no way to go beyond this number unless changing this setting. The data table used by Discover, EuiDataGrid, also doesn't provide ways to allow the incremental loading of data. There's no quick way to change this limitation, but what could be done is to use the UI that's currently telling users, that they have reached the limit, to provide a link allowing them to load more. So behind the scenes, we could usesearch_afterof ES search (https://www.elastic.co/guide/en/elasticsearch/reference/current/paginate-search-results.html#search-after) to append more documents to the current search result. So basically it could look like this:Before:
After:
The drawback of this approach: All documents, in this case, would be kept in the Browsers Memory, with adding more and more documents, the browser / host system might become unresponsive.
The text was updated successfully, but these errors were encountered: