[Fleet] Improve performance of Fleet setup #102219

afgomez · 2021-06-15T13:46:13Z

Summary

Closes #96026. See the issue's comments for details of what we have tried and measured.

This PR attempts to improve the setup time in Fleet. We are doing two things for it:

Remove endpoint from the list of default packages. The package is not in any of the default policies, so it can be safely removed.

Some functionality depends on the endpoint package being auto-updated, so we have introduced a new constant AUTO_UPDATE_PACKAGES with endpoint in it. Fleet will ensure the packages in this list will be updated to the last version, even if they are not installed by default.

Tweak the initial spinner, giving the user more information than just an empty loader.

Currently the loader is briefly shown even if Fleet is already set up, so we cannot do much regarding the text of that screen. We would need to first change how that loader is shown, and make it dependent of the actual setup status.

Testing

To test the new auto-update,

Boot a clean ES with yarn es snapshot -E xpack.security.authc.api_key.enabled=true
specify an old version on the endpoint package in your kibana.dev.yml

xpack.fleet.packages:
  - name: endpoint
    version: 0.18.0

Verify in http://localhost:5601/app/integrations that the version is installed.
without restarting ES, remove the custom version from kibana.dev.yml
Verify that the latest version is installed.

Checklist

Delete any items that are not applicable to this PR.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support

elasticmachine · 2021-06-15T13:46:16Z

Pinging @elastic/fleet (Feature:Fleet)

elasticmachine · 2021-06-15T13:46:16Z

Pinging @elastic/fleet (Team:Fleet)

jen-huang · 2021-06-15T17:53:12Z

As discussed with the Endpoint team, let's remove Endpoint from the list of packages that are auto-installed, but allow it to be auto-upgraded when visiting Fleet. I reviewed our current code here for default vs required packages, but their meaning doesn't suit our needs:

kibana/x-pack/plugins/fleet/common/constants/epm.ts

Line 14 in 6e0aed7

export const requiredPackages = {

I think we will need to create a third list:

"Default packages" = packages that are auto-installed: system, fleet_server, elastic_agent
"Required packages" = packages that cannot be uninstalled: same as above
"Auto upgrade packages" = packages that are auto-upgraded: default packages + endpoint

Default packages are presently auto-upgraded as well. With the creation of the third list, we'll have to adjust the logic so that auto-install acts on the list of default packages, and auto-upgrade acts on the list of auto-upgrade packages.

What do you think?

afgomez · 2021-06-16T08:02:21Z

I think we will need to create a third list:

"Default packages" = packages that are auto-installed: system, fleet_server, elastic_agent
"Required packages" = packages that cannot be uninstalled: same as above
"Auto upgrade packages" = packages that are auto-upgraded: default packages + endpoint

Makes sense, but I would collapse the first two lists into one. If a package cannot be uninstalled it needs to be auto-installed. We need to also consider how this should work when the user specifies the default packages/policies in their kibana.yaml.

I'll play with it and push something today

We want to keep the assumption that the lists contain the same packages only in `epm/constants.ts`

jen-huang · 2021-06-16T15:06:49Z

Makes sense, but I would collapse the first two lists into one. If a package cannot be uninstalled it needs to be auto-installed.

I do agree with this logic, but per some offline discussion we felt it would be safer to mimic the existing functionality of the Endpoint package as much as possible for this iteration. So the Endpoint package needs to:

Not be installed by default (changed behavior)
If installed, be auto-upgraded (same behavior)
If installed, not allow it to be uninstalled (same behavior)

jen-huang

I benchmarked /setup in this PR against main and saw ~6 seconds reduced loading time (20% reduction!) 🔥 I tested the non-auto-install, but auto-upgrade nature of Endpoint package and it seems to work well too.

There is probably more we could optimize in the actual install process to reduce the time even more, but I'm very happy to get this PR in as it is a significant improvement.

jen-huang · 2021-06-16T22:47:27Z

x-pack/plugins/fleet/public/components/fleet_setup_loading.tsx

+      </h2>
+    }
+    titleSize="m"
+    body={<EuiLoadingSpinner size="xl" />}


after #101334 is merged, we can take advantage of the new empty prompt loading pattern that was recently added to EUI which looks a bit nicer. not a blocker for this PR, we can come back to adjust it

I wanted to use that, but Kibana doesn't use yet the right version of EUI. Let's keep track of it

…etup

jonathan-buttner

Changes look good, we'll just need the endpoint tests to install the endpoint package as part of their setup and they should pass 👍

jonathan-buttner · 2021-06-17T13:52:12Z

x-pack/plugins/fleet/common/constants/epm.ts

-  FleetServer: FLEET_SERVER_PACKAGE,
-} as const;
+/*
+ Package rules:


Thanks for the table! Does requiredPackages just mean that they are not removable? I think that's what I'm seeing from the code. Maybe we should change it to unremovable or something. required makes me think that it must be installed by just reading the name. The table helps clarify it though 😄

…etup

jen-huang · 2021-06-17T15:07:04Z

@jonathan-buttner Thanks for testing! I made the Endpoint package be installed by default on tests by passing it in start options (b400b9d). Hopefully that approach works for you?

jonathan-buttner · 2021-06-17T18:19:46Z

@jonathan-buttner Thanks for testing! I made the Endpoint package be installed by default on tests by passing it in start options (b400b9d). Hopefully that approach works for you?

Sounds good to me!

kibanamachine · 2021-06-17T18:27:25Z

💚 Build Succeeded

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`fleet`	469	470	+1

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`fleet`	1011	1013	+2

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`fleet`	677.9KB	678.6KB	+721.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`fleet`	437.5KB	438.7KB	+1.2KB

Unknown metric groups

API count

id	before	after	diff
`fleet`	1102	1104	+2

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @afgomez

* Remove endpoint from the default packages * Change the default spinner for the initial load * Export fleet endpoint package as a constant * Use constants for special packages * Simplify type signature of `isRequiredPackage` * Remove unused types * Simplify required and default package definitions * Treat REQUIRED_PACKAGES as independent from DEFAULT_PACKAGES We want to keep the assumption that the lists contain the same packages only in `epm/constants.ts` * Install all default packages, not only the required ones * Document the purpose of each package list * Handle auto-update for non-default packages * Make `endpoint` non-removable * Make endpoint package be installed by default in tests * Rename requiredPackages to unremovablePackages * Fix type check * Add Endpoint to be installed by default on Fleet tests too Co-authored-by: Jen Huang <its.jenetic@gmail.com>

kibanamachine · 2021-06-17T19:01:36Z

💚 Backport successful

Status	Branch	Result
✅	7.x

This backport PR will be merged automatically after passing CI.

* Remove endpoint from the default packages * Change the default spinner for the initial load * Export fleet endpoint package as a constant * Use constants for special packages * Simplify type signature of `isRequiredPackage` * Remove unused types * Simplify required and default package definitions * Treat REQUIRED_PACKAGES as independent from DEFAULT_PACKAGES We want to keep the assumption that the lists contain the same packages only in `epm/constants.ts` * Install all default packages, not only the required ones * Document the purpose of each package list * Handle auto-update for non-default packages * Make `endpoint` non-removable * Make endpoint package be installed by default in tests * Rename requiredPackages to unremovablePackages * Fix type check * Add Endpoint to be installed by default on Fleet tests too Co-authored-by: Jen Huang <its.jenetic@gmail.com> Co-authored-by: Alejandro Fernández Gómez <alejandro.fernandez@elastic.co> Co-authored-by: Jen Huang <its.jenetic@gmail.com>

…ets-tab * 'master' of github.com:elastic/kibana: (93 commits) [ML] Remove blank job definition as it is unused and out-of-sync with Elasticsearch (elastic#102506) [Lens] Fix wrong error detection on transition to Top values operation (elastic#102384) [ML] Anomaly detection job custom_settings improvements (elastic#102099) [Cases] Route: Get all alerts attach to a case (elastic#101878) Fixes wrong list exception type when creating endpoint event filters list (elastic#102522) remove search bar that's not working yet (elastic#102550) Migrated Ingest Node Pipeline Functional Tests to use test_user (elastic#102409) [Maps] clean up feature editing name space to avoid conflicts with layer settings editing (elastic#102516) [canvas] Refactor Storybook from bespoke to standard configuration (elastic#101962) [Security Solution] adds wrapSequences method (RAC) (elastic#102106) [FTR] Stabilize SSLP functional tests (elastic#102553) [K8] Added `Inter` font files for new theme (elastic#102359) [Workplace Search] Convert Groups pages to new page template (elastic#102449) [DOC] Add experimental disclaimer to rollup jobs (elastic#95624) [Security Solution][Endpoint] Suppress some of the jest console.error noise created by endpoint list middelware (elastic#102535) [Fleet] Improve performance of Fleet setup (elastic#102219) [Alerting] Add event log entry when a rule starts executing (elastic#102001) [Fleet] Update docker image of registry used in integration tests (elastic#101911) [Asset Management] Osquery telemetry updates (elastic#100754) Converts saved object tagging to new management layout (elastic#102284) ... # Conflicts: # x-pack/plugins/fleet/kibana.json

dikshachauhan-qasource · 2021-06-23T07:57:32Z

Hi @jen-huang

We have validated this PR on 7.14 Snapshot build as per test steps mentioned in ticket summary and found changes are working fine.

Tested Endpoint package gets auto-update.

specify an old version on the endpoint package in your kibana.dev.yml
xpack.fleet.packages:
  - name: endpoint
    version: 0.18.0
Verify in http://localhost:5601/app/integrations that the version is installed.
without restarting ES, remove the custom version from kibana.dev.yml
Verify that the latest version is installed.

Followed above steps and observed it is working fine.

Screenshot:

Updated Loader shows up while fleet is uploading.

Screenshot:

Build details:

BUILD 41896
COMMIT e26582638988179d134e77e59b66ed8f982ab064

cc @EricDavisX

Please let us know if any further tests are required.

Thanks
QAS

Alejandro Fernández Gómez added 2 commits June 15, 2021 11:36

Remove endpoint from the default packages

0bd33ac

Change the default spinner for the initial load

a9d1051

afgomez requested a review from a team as a code owner June 15, 2021 13:46

afgomez self-assigned this Jun 15, 2021

nchaulet mentioned this pull request Jun 15, 2021

[Detections] Adds automatic updating for Prebuilt Security Detection Rules package #101846

Merged

2 tasks

Alejandro Fernández Gómez added 8 commits June 16, 2021 10:11

Export fleet endpoint package as a constant

dd27708

Use constants for special packages

affa5b9

Simplify type signature of isRequiredPackage

8cf78d6

Remove unused types

9e1026e

Simplify required and default package definitions

87560af

Treat REQUIRED_PACKAGES as independent from DEFAULT_PACKAGES

ddf3633

We want to keep the assumption that the lists contain the same packages only in `epm/constants.ts`

Install all default packages, not only the required ones

825bbf1

Document the purpose of each package list

3f85ec2

afgomez requested a review from a team as a code owner June 16, 2021 10:14

Handle auto-update for non-default packages

36cb102

Make endpoint non-removable

54720f2

jen-huang approved these changes Jun 16, 2021

View reviewed changes

Merge remote-tracking branch 'upstream/master' into 96026-ftl-fleet-s…

a69b64c

…etup

jonathan-buttner reviewed Jun 17, 2021

View reviewed changes

jen-huang added 3 commits June 17, 2021 07:57

Make endpoint package be installed by default in tests

b400b9d

Rename requiredPackages to unremovablePackages

e229c7d

Merge remote-tracking branch 'upstream/master' into 96026-ftl-fleet-s…

3e3a2dc

…etup

jen-huang added 2 commits June 17, 2021 08:24

Fix type check

7142778

Add Endpoint to be installed by default on Fleet tests too

1e46532

jonathan-buttner approved these changes Jun 17, 2021

View reviewed changes

jen-huang merged commit cd5cd65 into elastic:master Jun 17, 2021

kibanamachine mentioned this pull request Jun 17, 2021

[7.x] [Fleet] Improve performance of Fleet setup (#102219) #102559

Merged

mdelapenya mentioned this pull request Jun 21, 2021

No datastreams are listed on Fleet elastic/e2e-testing#1274

Closed

jonathan-buttner mentioned this pull request Jun 23, 2021

[Security Solution] Updating generator to install/update the endpoint package during setup #103094

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fleet] Improve performance of Fleet setup #102219

[Fleet] Improve performance of Fleet setup #102219

afgomez commented Jun 15, 2021 •

edited

Loading

elasticmachine commented Jun 15, 2021

elasticmachine commented Jun 15, 2021

jen-huang commented Jun 15, 2021

afgomez commented Jun 16, 2021

jen-huang commented Jun 16, 2021

jen-huang left a comment

jen-huang Jun 16, 2021

afgomez Jun 22, 2021

jonathan-buttner left a comment

jonathan-buttner Jun 17, 2021

jen-huang commented Jun 17, 2021

jonathan-buttner commented Jun 17, 2021

kibanamachine commented Jun 17, 2021

API count

kibanamachine commented Jun 17, 2021

dikshachauhan-qasource commented Jun 23, 2021

[Fleet] Improve performance of Fleet setup #102219

[Fleet] Improve performance of Fleet setup #102219

Conversation

afgomez commented Jun 15, 2021 • edited Loading

Summary

Testing

Checklist

elasticmachine commented Jun 15, 2021

elasticmachine commented Jun 15, 2021

jen-huang commented Jun 15, 2021

afgomez commented Jun 16, 2021

jen-huang commented Jun 16, 2021

jen-huang left a comment

Choose a reason for hiding this comment

jen-huang Jun 16, 2021

Choose a reason for hiding this comment

afgomez Jun 22, 2021

Choose a reason for hiding this comment

jonathan-buttner left a comment

Choose a reason for hiding this comment

jonathan-buttner Jun 17, 2021

Choose a reason for hiding this comment

jen-huang commented Jun 17, 2021

jonathan-buttner commented Jun 17, 2021

kibanamachine commented Jun 17, 2021

💚 Build Succeeded

Metrics [docs]

Module Count

Public APIs missing comments

Async chunks

Page load bundle

API count

History

kibanamachine commented Jun 17, 2021

💚 Backport successful

dikshachauhan-qasource commented Jun 23, 2021

afgomez commented Jun 15, 2021 •

edited

Loading