Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[7.x] [Security Solution][Detections] Adoption telemetry (#71102) #71504

Merged
merged 1 commit into from
Jul 13, 2020
Merged

[7.x] [Security Solution][Detections] Adoption telemetry (#71102) #71504

merged 1 commit into from
Jul 13, 2020

Conversation

rylnd
Copy link
Contributor

@rylnd rylnd commented Jul 13, 2020

Backports the following commits to 7.x:

@rylnd @elasticmachine
[Security Solution][Detections] Adoption telemetry (elastic#71102)
4d3ca34 
* style: sort plugin interface

* WIP: UsageCollector for Security Adoption

This uses ML and raw ES calls to query our ML Jobs and Rules, and parse
them into a format to be consumed by telemetry.

Still to come:
* initialization
* tests

* Initialize usage collectors during plugin setup

* Rename usage key

The service seems to convert colons to underscores, so let's just use an
underscure.

* Collector is ready if we have a kibana index

* Refactor collector to generate options in a function

This allows us to test our adherence to the collector API, focusing
particularly on the fetch function.

* Refactor usage collector in anticipation of endpoint data

We're going to have our usage data under one key corresponding to the
app, so this nests the existing data under a 'detections' key while
allowing another fetching function to be plugged into the
main collector under a separate key.

* Update our collector to satisfy telemetry tooling

* inlines collector options
* inlines schema object
* makes DetectionsUsage an interface instead of a type alias

* Extracts telemetry mappings via scripts/telemetry_extract

* Refactor detections usage logic to perform one loop instead of two

We were previously performing two loops over each set of data: one to
format it down to just the data we need, and another to convert that
into usage data. We now perform both steps within a single loop.

* Refactor detections telemetry to be nested

* Extract new nested detections telemetry mappings

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
@rylnd rylnd added the backport label Jul 13, 2020
@kibanamachine
Copy link
Contributor

💛 Build succeeded, but was flaky

Test Failures

Kibana Pipeline / kibana-oss-agent / Firefox UI Functional Tests.test/functional/apps/visualize/input_control_vis/chained_controls·js.visualize app input controls chained controls should disable child control when parent control is not set

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has failed 2 times on tracked branches: https://github.com/elastic/kibana/issues/68472

[00:00:00]       │
[00:13:15]         └-: visualize app
[00:13:15]           └-> "before all" hook
[00:13:15]           └-> "before all" hook
[00:13:15]             │ debg Starting visualize before method
[00:13:15]             │ info [logstash_functional] Loading "mappings.json"
[00:13:15]             │ info [logstash_functional] Loading "data.json.gz"
[00:13:15]             │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-debian-tests-xl-1594664830414461085] [logstash-2015.09.22] creating index, cause [api], templates [], shards [1]/[0]
[00:13:15]             │ info [o.e.c.r.a.AllocationService] [kibana-ci-immutable-debian-tests-xl-1594664830414461085] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[logstash-2015.09.22][0]]]).
[00:13:15]             │ info [logstash_functional] Created index "logstash-2015.09.22"
[00:13:15]             │ debg [logstash_functional] "logstash-2015.09.22" settings {"index":{"analysis":{"analyzer":{"url":{"max_token_length":"1000","tokenizer":"uax_url_email","type":"standard"}}},"number_of_replicas":"0","number_of_shards":"1"}}
[00:13:15]             │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-debian-tests-xl-1594664830414461085] [logstash-2015.09.20] creating index, cause [api], templates [], shards [1]/[0]
[00:13:15]             │ info [o.e.c.r.a.AllocationService] [kibana-ci-immutable-debian-tests-xl-1594664830414461085] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[logstash-2015.09.20][0]]]).
[00:13:15]             │ info [logstash_functional] Created index "logstash-2015.09.20"
[00:13:15]             │ debg [logstash_functional] "logstash-2015.09.20" settings {"index":{"analysis":{"analyzer":{"url":{"max_token_length":"1000","tokenizer":"uax_url_email","type":"standard"}}},"number_of_replicas":"0","number_of_shards":"1"}}
[00:13:15]             │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-debian-tests-xl-1594664830414461085] [logstash-2015.09.21] creating index, cause [api], templates [], shards [1]/[0]
[00:13:15]             │ info [o.e.c.r.a.AllocationService] [kibana-ci-immutable-debian-tests-xl-1594664830414461085] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[logstash-2015.09.21][0]]]).
[00:13:15]             │ info [logstash_functional] Created index "logstash-2015.09.21"
[00:13:15]             │ debg [logstash_functional] "logstash-2015.09.21" settings {"index":{"analysis":{"analyzer":{"url":{"max_token_length":"1000","tokenizer":"uax_url_email","type":"standard"}}},"number_of_replicas":"0","number_of_shards":"1"}}
[00:13:25]             │ info progress: 13139
[00:13:26]             │ info [logstash_functional] Indexed 4633 docs into "logstash-2015.09.22"
[00:13:26]             │ info [logstash_functional] Indexed 4757 docs into "logstash-2015.09.20"
[00:13:26]             │ info [logstash_functional] Indexed 4614 docs into "logstash-2015.09.21"
[00:13:26]             │ info [long_window_logstash] Loading "mappings.json"
[00:13:26]             │ info [long_window_logstash] Loading "data.json.gz"
[00:13:26]             │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-debian-tests-xl-1594664830414461085] [long-window-logstash-0] creating index, cause [api], templates [], shards [1]/[0]
[00:13:26]             │ info [o.e.c.r.a.AllocationService] [kibana-ci-immutable-debian-tests-xl-1594664830414461085] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[long-window-logstash-0][0]]]).
[00:13:26]             │ info [long_window_logstash] Created index "long-window-logstash-0"
[00:13:26]             │ debg [long_window_logstash] "long-window-logstash-0" settings {"index":{"analysis":{"analyzer":{"makelogs_url":{"max_token_length":"1000","tokenizer":"uax_url_email","type":"standard"}}},"number_of_replicas":"0","number_of_shards":"1"}}
[00:13:36]             │ info progress: 10536
[00:13:40]             │ info [long_window_logstash] Indexed 14005 docs into "long-window-logstash-0"
[00:13:40]             │ info [visualize] Loading "mappings.json"
[00:13:40]             │ info [visualize] Loading "data.json"
[00:13:40]             │ info [o.e.c.m.MetadataDeleteIndexService] [kibana-ci-immutable-debian-tests-xl-1594664830414461085] [.kibana_2/jGGnPhaHSKqfr7s1I6pNcg] deleting index
[00:13:40]             │ info [o.e.c.m.MetadataDeleteIndexService] [kibana-ci-immutable-debian-tests-xl-1594664830414461085] [.kibana_1/XmnJ-xSuQdim5XegryYSdg] deleting index
[00:13:40]             │ info [visualize] Deleted existing index [".kibana_2",".kibana_1"]
[00:13:40]             │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-debian-tests-xl-1594664830414461085] [.kibana] creating index, cause [api], templates [], shards [1]/[1]
[00:13:40]             │ info [visualize] Created index ".kibana"
[00:13:40]             │ debg [visualize] ".kibana" settings {"index":{"number_of_replicas":"1","number_of_shards":"1"}}
[00:13:40]             │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-debian-tests-xl-1594664830414461085] [.kibana/CM-uyuVOTnO9Rrukls65MA] update_mapping [_doc]
[00:13:40]             │ info [visualize] Indexed 12 docs into ".kibana"
[00:13:40]             │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-debian-tests-xl-1594664830414461085] [.kibana/CM-uyuVOTnO9Rrukls65MA] update_mapping [_doc]
[00:13:40]             │ debg Migrating saved objects
[00:13:40]             │ proc [kibana]   log   [19:00:19.207] [info][savedobjects-service] Creating index .kibana_2.
[00:13:40]             │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-debian-tests-xl-1594664830414461085] [.kibana_2] creating index, cause [api], templates [], shards [1]/[1]
[00:13:40]             │ info [o.e.c.r.a.AllocationService] [kibana-ci-immutable-debian-tests-xl-1594664830414461085] updating number_of_replicas to [0] for indices [.kibana_2]
[00:13:40]             │ proc [kibana]   log   [19:00:19.253] [info][savedobjects-service] Reindexing .kibana to .kibana_1
[00:13:40]             │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-debian-tests-xl-1594664830414461085] [.kibana_1] creating index, cause [api], templates [], shards [1]/[1]
[00:13:40]             │ info [o.e.c.r.a.AllocationService] [kibana-ci-immutable-debian-tests-xl-1594664830414461085] updating number_of_replicas to [0] for indices [.kibana_1]
[00:13:40]             │ info [o.e.t.LoggingTaskListener] [kibana-ci-immutable-debian-tests-xl-1594664830414461085] 9220 finished with response BulkByScrollResponse[took=33.3ms,timed_out=false,sliceId=null,updated=0,created=12,deleted=0,batches=1,versionConflicts=0,noops=0,retries=0,throttledUntil=0s,bulk_failures=[],search_failures=[]]
[00:13:40]             │ info [o.e.c.m.MetadataDeleteIndexService] [kibana-ci-immutable-debian-tests-xl-1594664830414461085] [.kibana/CM-uyuVOTnO9Rrukls65MA] deleting index
[00:13:40]             │ proc [kibana]   log   [19:00:19.594] [info][savedobjects-service] Migrating .kibana_1 saved objects to .kibana_2
[00:13:41]             │ proc [kibana]   log   [19:00:19.609] [error][savedobjects-service] Error: Unable to migrate the corrupt Saved Object document index-pattern:test_index*. To prevent Kibana from performing a migration on every restart, please delete or fix this document by ensuring that the namespace and type in the document's id matches the values in the namespace and type fields.
[00:13:41]             │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-debian-tests-xl-1594664830414461085] [.kibana_2/ti3gpuVWSYyEUqI6Sd-X1Q] update_mapping [_doc]
[00:13:41]             │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-debian-tests-xl-1594664830414461085] [.kibana_2/ti3gpuVWSYyEUqI6Sd-X1Q] update_mapping [_doc]
[00:13:41]             │ proc [kibana]   log   [19:00:19.696] [info][savedobjects-service] Pointing alias .kibana to .kibana_2.
[00:13:41]             │ proc [kibana]   log   [19:00:19.754] [info][savedobjects-service] Finished in 549ms.
[00:13:41]             │ debg applying update to kibana config: {"accessibility:disableAnimations":true,"dateFormat:tz":"UTC"}
[00:13:41]             │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-debian-tests-xl-1594664830414461085] [.kibana_2/ti3gpuVWSYyEUqI6Sd-X1Q] update_mapping [_doc]
[00:13:42]             │ debg replacing kibana config doc: {"defaultIndex":"logstash-*","format:bytes:defaultPattern":"0,0.[000]b"}
[00:13:43]           └-: 
[00:13:43]             └-> "before all" hook
[00:13:43]             └-: input controls
[00:13:43]               └-> "before all" hook
[00:13:43]               └-: chained controls
[00:13:43]                 └-> "before all" hook
[00:13:43]                 └-> "before all" hook
[00:13:43]                   │ debg navigating to visualize url: http://localhost:6111/app/visualize#/
[00:13:43]                   │ debg navigate to: http://localhost:6111/app/visualize#/
[00:13:43]                   │ debg browser[log] "^ A single error about an inline script not firing due to content security policy is expected!"
[00:13:43]                   │ debg ... sleep(700) start
[00:13:44]                   │ debg ... sleep(700) end
[00:13:44]                   │ debg returned from get, calling refresh
[00:13:45]                   │ debg browser[info] "INFO: 2020-07-13T19:00:24Z\n  Adding connection to http://localhost:6111/elasticsearch\n\n"
[00:13:45]                   │ERROR browser[error] (new TypeError("NetworkError when attempting to fetch resource.", ""))
[00:13:45]                   │ debg browser[log] "^ A single error about an inline script not firing due to content security policy is expected!"
[00:13:46]                   │ debg currentUrl = http://localhost:6111/app/visualize#/
[00:13:46]                   │          appUrl = http://localhost:6111/app/visualize#/
[00:13:46]                   │ debg TestSubjects.find(kibanaChrome)
[00:13:46]                   │ debg Find.findByCssSelector('[data-test-subj="kibanaChrome"]') with timeout=60000
[00:13:47]                   │ debg browser[info] "INFO: 2020-07-13T19:00:26Z\n  Adding connection to http://localhost:6111/elasticsearch\n\n"
[00:13:48]                   │ debg ... sleep(501) start
[00:13:48]                   │ debg ... sleep(501) end
[00:13:48]                   │ debg in navigateTo url = http://localhost:6111/app/visualize#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-15m,to:now))
[00:13:48]                   │ debg --- retry.try error: URL changed, waiting for it to settle
[00:13:49]                   │ debg ... sleep(501) start
[00:13:49]                   │ debg ... sleep(501) end
[00:13:49]                   │ debg in navigateTo url = http://localhost:6111/app/visualize#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-15m,to:now))
[00:13:49]                   │ debg TestSubjects.exists(statusPageContainer)
[00:13:49]                   │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="statusPageContainer"]') with timeout=2500
[00:13:52]                   │ debg --- retry.tryForTime error: [data-test-subj="statusPageContainer"] is not displayed
[00:13:52]                   │ debg TestSubjects.click(visListingTitleLink-chained-input-control)
[00:13:52]                   │ debg Find.clickByCssSelector('[data-test-subj="visListingTitleLink-chained-input-control"]') with timeout=20000
[00:13:52]                   │ debg Find.findByCssSelector('[data-test-subj="visListingTitleLink-chained-input-control"]') with timeout=20000
[00:13:52]                   │ debg isGlobalLoadingIndicatorVisible
[00:13:52]                   │ debg TestSubjects.exists(globalLoadingIndicator)
[00:13:52]                   │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="globalLoadingIndicator"]') with timeout=1500
[00:13:53]                   │ debg --- retry.tryForTime error: [data-test-subj="globalLoadingIndicator"] is not displayed
[00:13:53]                   │ debg TestSubjects.exists(globalLoadingIndicator-hidden)
[00:13:53]                   │ debg Find.existsByCssSelector('[data-test-subj="globalLoadingIndicator-hidden"]') with timeout=100000
[00:13:54]                 └-> should disable child control when parent control is not set
[00:13:54]                   └-> "before each" hook: global before each
[00:13:54]                   │ debg comboBox.getOptionsList, comboBoxSelector: listControlSelect0
[00:13:54]                   │ debg TestSubjects.find(listControlSelect0)
[00:13:54]                   │ debg Find.findByCssSelector('[data-test-subj="listControlSelect0"]') with timeout=10000
[00:14:04]                   │ info Taking screenshot "/dev/shm/workspace/kibana/test/functional/screenshots/failure/visualize app  input controls chained controls should disable child control when parent control is not set.png"
[00:14:05]                   │ info Current URL is: http://localhost:6111/app/visualize#/edit/68305470-87bc-11e9-a991-3b492a7c3e09?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-15m,to:now))&_a=(filters:!(),linked:!f,query:(language:kuery,query:%27%27),uiState:(),vis:(aggs:!(),params:(controls:!((fieldName:geo.src,id:%271559757816862%27,indexPattern:%27logstash-*%27,label:%27%27,options:(dynamicOptions:!t,multiselect:!t,order:desc,size:5,type:terms),parent:%27%27,type:list),(fieldName:clientip,id:%271559757836347%27,indexPattern:%27logstash-*%27,label:%27%27,options:(dynamicOptions:!t,multiselect:!t,order:desc,size:5,type:terms),parent:%271559757816862%27,type:list)),pinFilters:!f,updateFiltersOnChange:!f,useTimeFilter:!f),title:%27chained%20input%20control%27,type:input_control_vis))
[00:14:05]                   │ info Saving page source to: /dev/shm/workspace/kibana/test/functional/failure_debug/html/visualize app  input controls chained controls should disable child control when parent control is not set.html
[00:14:05]                   └- ✖ fail: visualize app  input controls chained controls should disable child control when parent control is not set
[00:14:05]                   │      TimeoutError: Waiting for element to be located By(css selector, [data-test-subj="listControlSelect0"])
[00:14:05]                   │ Wait timed out after 10020ms
[00:14:05]                   │       at /dev/shm/workspace/kibana/node_modules/selenium-webdriver/lib/webdriver.js:842:17
[00:14:05]                   │       at process._tickCallback (internal/process/next_tick.js:68:7)
[00:14:05]                   │ 
[00:14:05]                   │

Stack Trace

{ TimeoutError: Waiting for element to be located By(css selector, [data-test-subj="listControlSelect0"])
Wait timed out after 10020ms
    at /dev/shm/workspace/kibana/node_modules/selenium-webdriver/lib/webdriver.js:842:17
    at process._tickCallback (internal/process/next_tick.js:68:7) name: 'TimeoutError', remoteStacktrace: '' }

Build metrics

✅ unchanged

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@rylnd rylnd merged commit 6e1aaad into elastic:7.x Jul 13, 2020
@rylnd rylnd deleted the backport/7.x/pr-71102 branch July 13, 2020 20:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
backport
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@rylnd @kibanamachine