[Asset Management] Osquery telemetry updates #100754

lykkin · 2021-05-27T04:13:31Z

Summary

Adds osquery app usage metrics to the kibana telemetry mapping
Adds support to collect osquery app usage metrics

TODO in a follow on PR:
Get beat payload size + query duration metrics.

elasticmachine · 2021-06-04T05:51:26Z

Pinging @elastic/security-asset-management (Team:Asset Management)

lykkin · 2021-06-04T05:53:57Z

@elasticmachine merge upstream

lykkin · 2021-06-04T14:51:34Z

@elasticmachine merge upstream

lukeelmers

Schema updates LGTM.

nit: Would recommend some unit tests for your usage collectors, as we've uncovered some legit bugs with them in the past (especially in cases where you are relying on stuff happening to determine isReady). Some examples in src/plugins/kibana_usage_collection/server/collectors

x-pack/plugins/osquery/server/usage/fetchers.ts

pjhampton

🌔 🚀 ✨ LGTM ✨ 🚀 🌔

Just some questions but otherwise, looks good.

pjhampton · 2021-06-07T10:15:34Z

x-pack/plugins/osquery/server/routes/usage/recorder.ts

+) {
+  const metric = await soClient.get<CounterValue>(usageMetricSavedObjectType, route);
+  metric.attributes[key] += increment;
+  await soClient.update(usageMetricSavedObjectType, route, metric.attributes);


Does this ever flush the state, or does it just increment forever?

this is a cumulative count. i'm not entirely certain about the life times of saved objects, what events do they get flushed in outside of clearing them manually? i might be able to do something like clearing the count on fetch, but it felt like a wash to me.

are there any situations where delta metrics would be better for this kind of thing?

Thanks for bearing with me. I have been OOO for a couple of days :-)

what events do they get flushed in outside of clearing them manually?

They don't - unless a space is deleted AFAIK.

are there any situations where delta metrics would be better for this kind of thing?

Delta is maybe not a good word to describe it, but from what I have seen the usage collector runs every 24 hr and developers typically try to get a snapshot of the last 24 hours. That makes me think that clearing the count on read is what is needed. But I guess it only matters to people / systems consuming the telemetry. It's your call :- )

thanks for the context! i feel better about changing it knowing there's prior art, and knowing about the collection interval.

x-pack/plugins/osquery/server/usage/collector.ts

x-pack/plugins/osquery/server/usage/fetchers.ts

Bamieh

schema LGTM. I love the changes in this PR :elasticheart: other than the savedObjects client type casting LGTM.

x-pack/plugins/osquery/server/usage/collector.ts

lykkin · 2021-06-10T15:11:14Z

@elasticmachine merge upstream

lykkin · 2021-06-15T19:35:18Z

@elastic/kibana-core - i'm curious if the usage of saved objects in this pr would cause potential double counting of the stats being tracked in them when multiple kibana instances are being used for the same ES instance. are my worries misplaced?

pjhampton · 2021-06-16T09:01:18Z

@elastic/kibana-core - i'm curious if the usage of saved objects in this pr would cause potential double counting of the stats being tracked in them when multiple kibana instances are being used for the same ES instance. are my worries misplaced?

@Bamieh could give a more authoritative answer than me, but I don't believe the usage collector would run on each Kibana instance - I think it would be an available worker approach. This is because from my own experience I haven't seen duplicated telemetry records coming through the security pipelines.

Bamieh · 2021-06-17T08:47:49Z

@pjhampton @lykkin I'm gonna follow up sync about the PR just to double check I havent missed anything before giving a final OK.

I cant find areas where double counting is possible. We do have usage counters to replace this counting implementation and handle all these details instead of manually writing this logic here.

lykkin · 2021-06-17T16:49:55Z

@elasticmachine merge upstream

Bamieh · 2021-06-17T18:05:26Z

I've synced with Bryan and we're good to merge 👍

kibanamachine · 2021-06-17T18:53:25Z

💚 Build Succeeded

Metrics [docs]

Saved Objects .kibana field count

Every field in each saved object type adds overhead to Elasticsearch. Kibana needs to keep the total field count below Elasticsearch's default limit of 1000 fields. Only specify field mappings for the fields you wish to search on or query. See https://www.elastic.co/guide/en/kibana/master/development-plugin-saved-objects.html#_mappings

id	before	after	diff
`osquery-usage-metric`	-	3	+3

History

💚 Build #130516 succeeded 91d1b19
💛 Build #130384 was flaky bcb6138
💔 Build #130374 failed be2e6f0
💚 Build #129385 succeeded 6d50ffd
💚 Build #129205 succeeded 0f38288

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @lykkin

kibanamachine · 2021-06-17T18:56:11Z

💚 Backport successful

Status	Branch	Result
✅	7.x

This backport PR will be merged automatically after passing CI.

* first pass of basic osquery usage stats collection * updates, linting * updated exported metrics * clean up comments, add description fields to metric fields * reworked types * actually use the updated types * added tests around the route usage recoder functions * review comments * update aggregate types Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

* first pass of basic osquery usage stats collection * updates, linting * updated exported metrics * clean up comments, add description fields to metric fields * reworked types * actually use the updated types * added tests around the route usage recoder functions * review comments * update aggregate types Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Bryan Clement <bclement01@gmail.com>

…ets-tab * 'master' of github.com:elastic/kibana: (93 commits) [ML] Remove blank job definition as it is unused and out-of-sync with Elasticsearch (elastic#102506) [Lens] Fix wrong error detection on transition to Top values operation (elastic#102384) [ML] Anomaly detection job custom_settings improvements (elastic#102099) [Cases] Route: Get all alerts attach to a case (elastic#101878) Fixes wrong list exception type when creating endpoint event filters list (elastic#102522) remove search bar that's not working yet (elastic#102550) Migrated Ingest Node Pipeline Functional Tests to use test_user (elastic#102409) [Maps] clean up feature editing name space to avoid conflicts with layer settings editing (elastic#102516) [canvas] Refactor Storybook from bespoke to standard configuration (elastic#101962) [Security Solution] adds wrapSequences method (RAC) (elastic#102106) [FTR] Stabilize SSLP functional tests (elastic#102553) [K8] Added `Inter` font files for new theme (elastic#102359) [Workplace Search] Convert Groups pages to new page template (elastic#102449) [DOC] Add experimental disclaimer to rollup jobs (elastic#95624) [Security Solution][Endpoint] Suppress some of the jest console.error noise created by endpoint list middelware (elastic#102535) [Fleet] Improve performance of Fleet setup (elastic#102219) [Alerting] Add event log entry when a rule starts executing (elastic#102001) [Fleet] Update docker image of registry used in integration tests (elastic#101911) [Asset Management] Osquery telemetry updates (elastic#100754) Converts saved object tagging to new management layout (elastic#102284) ... # Conflicts: # x-pack/plugins/fleet/kibana.json

lykkin added release_note:enhancement v8.0.0 v7.14.0 Team:Asset Management Security Asset Management Team Feature:Osquery Security Solution Osquery feature auto-backport Deprecated - use backport:version if exact versions are needed labels May 27, 2021

lykkin self-assigned this May 27, 2021

lykkin force-pushed the osquery-telemetry branch from 21add4c to 266f58d Compare June 3, 2021 09:06

lykkin marked this pull request as ready for review June 4, 2021 05:51

lykkin requested review from a team as code owners June 4, 2021 05:51

patrykkopycinski requested a review from pjhampton June 4, 2021 13:39

lukeelmers approved these changes Jun 4, 2021

View reviewed changes

pjhampton reviewed Jun 7, 2021

View reviewed changes

x-pack/plugins/osquery/server/usage/fetchers.ts Outdated Show resolved Hide resolved

pjhampton approved these changes Jun 7, 2021

View reviewed changes

Bamieh approved these changes Jun 9, 2021

View reviewed changes

x-pack/plugins/osquery/server/usage/collector.ts Outdated Show resolved Hide resolved

lykkin added 8 commits June 9, 2021 23:40

first pass of basic osquery usage stats collection

adb5276

updates, linting

821eee4

updated exported metrics

cab8d8d

clean up comments, add description fields to metric fields

838ae1d

reworked types

d01db68

actually use the updated types

d6b904d

added tests around the route usage recoder functions

eb20cf8

review comments

be2e6f0

lykkin force-pushed the osquery-telemetry branch from 6d50ffd to be2e6f0 Compare June 10, 2021 07:02

update aggregate types

bcb6138

Merge branch 'master' into osquery-telemetry

91d1b19

Merge branch 'master' into osquery-telemetry

c7746d2

patrykkopycinski approved these changes Jun 17, 2021

View reviewed changes

lykkin enabled auto-merge (squash) June 17, 2021 17:14

lykkin merged commit 62f3a55 into elastic:master Jun 17, 2021

kibanamachine mentioned this pull request Jun 17, 2021

[7.x] [Asset Management] Osquery telemetry updates (#100754) #102557

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Asset Management] Osquery telemetry updates #100754

[Asset Management] Osquery telemetry updates #100754

lykkin commented May 27, 2021 •

edited

Loading

elasticmachine commented Jun 4, 2021

lykkin commented Jun 4, 2021

lykkin commented Jun 4, 2021

lukeelmers left a comment

pjhampton left a comment

pjhampton Jun 7, 2021

lykkin Jun 8, 2021

pjhampton Jun 14, 2021

lykkin Jun 14, 2021

Bamieh left a comment

lykkin commented Jun 10, 2021

lykkin commented Jun 15, 2021

pjhampton commented Jun 16, 2021

Bamieh commented Jun 17, 2021

lykkin commented Jun 17, 2021

Bamieh commented Jun 17, 2021

kibanamachine commented Jun 17, 2021

kibanamachine commented Jun 17, 2021

[Asset Management] Osquery telemetry updates #100754

[Asset Management] Osquery telemetry updates #100754

Conversation

lykkin commented May 27, 2021 • edited Loading

Summary

elasticmachine commented Jun 4, 2021

lykkin commented Jun 4, 2021

lykkin commented Jun 4, 2021

lukeelmers left a comment

Choose a reason for hiding this comment

pjhampton left a comment

Choose a reason for hiding this comment

pjhampton Jun 7, 2021

Choose a reason for hiding this comment

lykkin Jun 8, 2021

Choose a reason for hiding this comment

pjhampton Jun 14, 2021

Choose a reason for hiding this comment

lykkin Jun 14, 2021

Choose a reason for hiding this comment

Bamieh left a comment

Choose a reason for hiding this comment

lykkin commented Jun 10, 2021

lykkin commented Jun 15, 2021

pjhampton commented Jun 16, 2021

Bamieh commented Jun 17, 2021

lykkin commented Jun 17, 2021

Bamieh commented Jun 17, 2021

kibanamachine commented Jun 17, 2021

💚 Build Succeeded

Metrics [docs]

Saved Objects .kibana field count

History

kibanamachine commented Jun 17, 2021

💚 Backport successful

lykkin commented May 27, 2021 •

edited

Loading