Sort server-side in SavedObject export #55128

tylersmalley · 2020-01-16T21:57:55Z

Sorting/Aggregating by _id was removed in #52517

I am not seeing a reason yet as to why the sorting here is necessary as we never perform additional fetches of the data outside of exportSizeLimit which would require consistent sorting.

cc: @rudolf

elasticmachine · 2020-01-16T21:57:58Z

Pinging @elastic/kibana-platform (Team:Platform)

rudolf

I've asked Mike Cote to take a look as well, but my guess is this was only added to produce stable exports.

mikecote · 2020-01-17T12:48:27Z

The sort by _id was there to solve this issue #29747. Though for now since we're loading them all at the same time, we could just do so on the server side or drop the this feature overall.

rudolf · 2020-01-17T13:17:33Z

If this feature was requested by users we should rather keep it by doing a server-side sort. We would eventually like to stream these results to support huge exports but there's no immediate timeline for implementing that.

Sorting/Aggregating by _id was removed in elastic#52517 Retains sorting as requested by elastic#29747 Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>

Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>

tylersmalley · 2020-01-24T21:14:30Z

@elasticmachine merge upstream

tylersmalley · 2020-01-24T21:23:38Z

I have update to sort server-side.

I think doing this with ES does make more sense, but we would need another field to sort-by. We need to either copy _id to something with doc_values enabled, or find another field to sort by. I think finding another field would make sense but would be a breaking chance since the order would change. We should consider having a created_at field which is automatically populated which we could then sort by.

src/core/server/saved_objects/export/get_sorted_objects_for_export.ts

rudolf · 2020-01-26T21:10:10Z

We should consider having a created_at field which is automatically populated which we could then sort by.

This sounds like a good idea.

Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>

tylersmalley · 2020-01-27T20:35:45Z

Discussed with @mikecote who recommended we just sort the ES response in fetchObjectsToExport where the ES sort was removed. sortObjects is sorting for priority of dependencies, which will take precedence but still result in consistent results.

tylersmalley · 2020-01-27T21:38:26Z

src/core/server/saved_objects/export/get_sorted_objects_for_export.test.ts

@@ -399,6 +393,79 @@ describe('getSortedObjectsForExport()', () => {
    ).rejects.toThrowErrorMatchingInlineSnapshot(`"Can't export more than 1 objects"`);
  });

+  test('sorts objects within type', async () => {
+    savedObjectsClient.find.mockResolvedValueOnce({


There previously was only coverage when objects were provided, resulting in the bulk request - so this also has the benefit of including that path using find.

rudolf

Discussed with @mikecote who recommended we just sort the ES response in fetchObjectsToExport where the ES sort was removed. sortObjects is sorting for priority of dependencies, which will take precedence but still result in consistent results.

Ah I missed that 👍

src/core/server/saved_objects/export/get_sorted_objects_for_export.ts

Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>

kibanamachine · 2020-01-28T18:59:54Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 6b9becc

History

💚 Build #22831 succeeded 4935025
💚 Build #22490 succeeded 90daef0
💚 Build #20963 succeeded 6e17407

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

* master: (31 commits) [SIEM] Overview page feedback (elastic#56261) refactor (elastic#56131) [NP Cleanup] Remove ui/public/inspector (elastic#55677) [SIEM] [TIMELINE] Only add endpoint logo when on event.module === endgame (elastic#56263) Basic Functionality Alert List (elastic#55800) [SIEM] Fix filters on Hosts and Network page (elastic#56234) [SIEM] Adds ability to infer the newsfeed.enabled setting (elastic#56236) [SIEM][Detection Engine] critical blocker for updated rules [SIEM][Detection Engine] critical blocker, fixes ordering issue that causes rules to not run the first time [SIEM] Add link to endpoint app through reference.url (elastic#56211) [Metrics UI] Fixing title truncation in Metrics Explorer (elastic#55917) [SIEM] Put the notice for rules in comment block (elastic#56123) [SIEM][Detection Engine] critical blocker with the UI crashing Consistent timeouts for the Space onPostAuth interceptor tests (elastic#56158) Skip tests that depend on other skipped test [SIEM] [Detection Engine] Timestamps for rules (elastic#56197) Sort server-side in SavedObject export (elastic#55128) [Reporting] Document the 8.0 breaking changes (elastic#56187) Revert "[Monitoring] Change all configs to `monitoring.*`" (elastic#56214) add owners for es_archiver (elastic#56184) ...

tylersmalley · 2020-01-29T18:15:39Z

@mikecote, do you want this backported to 7.x - or just leave it at 8.0?

mikecote · 2020-01-29T18:17:13Z

@tylersmalley

@mikecote, do you want this backported to 7.x - or just leave it at 8.0?

I don't see why not, might as well backport this to 7.x.

Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>

Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

tylersmalley · 2020-02-10T22:15:43Z

👍 @mikecote, backported #57243

tylersmalley added Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc v8.0.0 release_note:skip Skip the PR/issue when compiling release notes labels Jan 16, 2020

tylersmalley requested a review from rudolf January 16, 2020 21:57

tylersmalley requested a review from a team as a code owner January 16, 2020 21:57

tylersmalley mentioned this pull request Jan 16, 2020

Testing unverified ES snapshot #54879

Closed

rudolf requested a review from mikecote January 17, 2020 09:13

rudolf approved these changes Jan 17, 2020

View reviewed changes

Sort SavedObjects server-side instead of by _id in ES

0c735e1

Sorting/Aggregating by _id was removed in elastic#52517 Retains sorting as requested by elastic#29747 Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>

tylersmalley force-pushed the remove-sort-by-id branch from 6e17407 to 0c735e1 Compare January 24, 2020 21:10

Adds comment

c821ba3

Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>

rudolf requested changes Jan 26, 2020

View reviewed changes

src/core/server/saved_objects/export/get_sorted_objects_for_export.ts Outdated Show resolved Hide resolved

tylersmalley changed the title ~~Removes sorting by _id in SavedObject export~~ Sort server-side in SavedObject export Jan 27, 2020

Tyler Smalley added 2 commits January 27, 2020 12:29

Sorts as part of fetchObjectsToExport

48883f9

Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>

Merge remote-tracking branch 'upstream/master' into remove-sort-by-id

4935025

tylersmalley force-pushed the remove-sort-by-id branch from 90daef0 to 4935025 Compare January 27, 2020 20:30

tylersmalley commented Jan 27, 2020

View reviewed changes

rudolf approved these changes Jan 28, 2020

View reviewed changes

src/core/server/saved_objects/export/get_sorted_objects_for_export.ts Outdated Show resolved Hide resolved

Cleanup object assignment

6b9becc

Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>

tylersmalley merged commit ff37dd1 into elastic:master Jan 28, 2020

tylersmalley mentioned this pull request Feb 10, 2020

[7.x] Sort server-side in SavedObject export (#55128) #57243

Merged

tylersmalley pushed a commit to tylersmalley/kibana that referenced this pull request Feb 10, 2020

Sort server-side in SavedObject export (elastic#55128)

4161479

Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>

tylersmalley pushed a commit that referenced this pull request Feb 10, 2020

Sort server-side in SavedObject export (#55128) (#57243)

b548737

Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

tylersmalley added the v7.7.0 label Mar 2, 2020

rudolf added the Feature:Saved Objects label Mar 15, 2021

rudolf mentioned this pull request Mar 15, 2021

Investigate saved objects import/export scalability #94552

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sort server-side in SavedObject export #55128

Sort server-side in SavedObject export #55128

tylersmalley commented Jan 16, 2020

elasticmachine commented Jan 16, 2020

rudolf left a comment

mikecote commented Jan 17, 2020

rudolf commented Jan 17, 2020

tylersmalley commented Jan 24, 2020

tylersmalley commented Jan 24, 2020

rudolf commented Jan 26, 2020

tylersmalley commented Jan 27, 2020

tylersmalley Jan 27, 2020 •

edited

Loading

rudolf left a comment

kibanamachine commented Jan 28, 2020

tylersmalley commented Jan 29, 2020

mikecote commented Jan 29, 2020

tylersmalley commented Feb 10, 2020

Sort server-side in SavedObject export #55128

Sort server-side in SavedObject export #55128

Conversation

tylersmalley commented Jan 16, 2020

elasticmachine commented Jan 16, 2020

rudolf left a comment

Choose a reason for hiding this comment

mikecote commented Jan 17, 2020

rudolf commented Jan 17, 2020

tylersmalley commented Jan 24, 2020

tylersmalley commented Jan 24, 2020

rudolf commented Jan 26, 2020

tylersmalley commented Jan 27, 2020

tylersmalley Jan 27, 2020 • edited Loading

Choose a reason for hiding this comment

rudolf left a comment

Choose a reason for hiding this comment

kibanamachine commented Jan 28, 2020

💚 Build Succeeded

History

tylersmalley commented Jan 29, 2020

mikecote commented Jan 29, 2020

tylersmalley commented Feb 10, 2020

tylersmalley Jan 27, 2020 •

edited

Loading