[7.x] [Endpoint] Policy Details integration with Ingest APIs (#61827)

[paul-tavares]

Backports the following commits to 7.x:

• [Endpoint] Policy Details integration with Ingest APIs ([Endpoint] Policy Details integration with Ingest APIs #61827)

[kibanamachine]

💛 Build succeeded, but was flaky

• continuous-integration/kibana-ci/pull-request
• Commit: 14ae478
• Flaky suites:
  • xpack-kibana-ciGroup1

---

Test Failures

Kibana Pipeline / kibana-xpack-agent / X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/security_and_spaces/tests/create_rules·ts.detection engine api security and spaces enabled create_rules creating rules should create a single Machine Learning rule

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has failed 5 times on tracked branches: https://github.com/elastic/kibana/issues/61995

[00:00:00]       │
[00:00:00]         └-: detection engine api security and spaces enabled
[00:00:00]           └-> "before all" hook
[00:00:13]           └-: create_rules
[00:00:13]             └-> "before all" hook
[00:00:13]             └-: creating rules
[00:00:13]               └-> "before all" hook
[00:00:13]               └-> should create a single rule with a rule_id
[00:00:13]                 └-> "before each" hook: global before each
[00:00:13]                 └-> "before each" hook
[00:00:13]                   │ info [o.e.x.i.a.TransportPutLifecycleAction] [kibana-ci-immutable-ubuntu-18-tests-xl-1585688074551984762] adding index lifecycle policy [.siem-signals-default]
[00:00:13]                   │ info [o.e.c.m.MetaDataIndexTemplateService] [kibana-ci-immutable-ubuntu-18-tests-xl-1585688074551984762] adding template [.siem-signals-default] for index patterns [.siem-signals-default-*]
[00:00:13]                   │ info [o.e.c.m.MetaDataCreateIndexService] [kibana-ci-immutable-ubuntu-18-tests-xl-1585688074551984762] [.siem-signals-default-000001] creating index, cause [api], templates [.siem-signals-default], shards [1]/[1], mappings [_doc]
[00:00:13]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-ubuntu-18-tests-xl-1585688074551984762] moving index [.siem-signals-default-000001] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [.siem-signals-default]
[00:00:14]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-ubuntu-18-tests-xl-1585688074551984762] moving index [.siem-signals-default-000001] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] in policy [.siem-signals-default]
[00:00:14]                 │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-ubuntu-18-tests-xl-1585688074551984762] moving index [.siem-signals-default-000001] from [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-follow-shard-tasks"}] in policy [.siem-signals-default]
[00:00:18]                 └- ✓ pass  (4.8s) "detection engine api security and spaces enabled create_rules creating rules should create a single rule with a rule_id"
[00:00:18]               └-> "after each" hook
[00:00:18]                 │ info [o.e.c.m.MetaDataDeleteIndexService] [kibana-ci-immutable-ubuntu-18-tests-xl-1585688074551984762] [.siem-signals-default-000001/dpY1t3QZSQi8aBIQlB9K1Q] deleting index
[00:00:19]                 │ info [o.e.c.m.MetaDataIndexTemplateService] [kibana-ci-immutable-ubuntu-18-tests-xl-1585688074551984762] removing template [.siem-signals-default]
[00:00:19]               └-> should create a single rule without an input index
[00:00:19]                 └-> "before each" hook: global before each
[00:00:19]                 └-> "before each" hook
[00:00:19]                   │ info [o.e.x.s.a.AuthenticationService] [kibana-ci-immutable-ubuntu-18-tests-xl-1585688074551984762] Authentication using apikey failed - api key has been invalidated
[00:00:19]                   │ proc [kibana]   log   [22:05:51.450] [error][plugins][siem] An error occurred during rule execution:
[00:00:19]                   │ proc [kibana] message: "[security_exception] missing authentication credentials for REST request [/auditbeat-*/_search?allow_no_indices=true&size=100&ignore_unavailable=true], with { header={ WWW-Authenticate={ 0="Bearer realm=\"security\"" & 1="ApiKey" & 2="Basic realm=\"security\" charset=\"UTF-8\"" } } }"
[00:00:19]                   │ proc [kibana] name: "Simple Rule Query"
[00:00:19]                   │ proc [kibana] id: "e7e644a1-1bac-4ec5-b4c5-541c67da54f4"
[00:00:19]                   │ proc [kibana] rule id: "rule-1"
[00:00:19]                   │ proc [kibana] signals index: ".siem-signals-default"
[00:00:19]                   │ info [o.e.x.s.a.AuthenticationService] [kibana-ci-immutable-ubuntu-18-tests-xl-1585688074551984762] Authentication using apikey failed - api key has been invalidated
[00:00:19]                   │ proc [kibana]   log   [22:05:51.464] [error][alerting][alerting][plugins][plugins] Executing Alert "e7e644a1-1bac-4ec5-b4c5-541c67da54f4" has resulted in Error: [security_exception] missing authentication credentials for REST request [/_security/user/_has_privileges], with { header={ WWW-Authenticate={ 0="Bearer realm=\"security\"" & 1="ApiKey" & 2="Basic realm=\"security\" charset=\"UTF-8\"" } } }
[00:00:19]                   │ info [o.e.x.i.a.TransportPutLifecycleAction] [kibana-ci-immutable-ubuntu-18-tests-xl-1585688074551984762] adding index lifecycle policy [.siem-signals-default]
[00:00:19]                   │ info [o.e.c.m.MetaDataIndexTemplateService] [kibana-ci-immutable-ubuntu-18-tests-xl-1585688074551984762] adding template [.siem-signals-default] for index patterns [.siem-signals-default-*]
[00:00:19]                   │ info [o.e.c.m.MetaDataCreateIndexService] [kibana-ci-immutable-ubuntu-18-tests-xl-1585688074551984762] [.siem-signals-default-000001] creating index, cause [api], templates [.siem-signals-default], shards [1]/[1], mappings [_doc]
[00:00:19]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-ubuntu-18-tests-xl-1585688074551984762] moving index [.siem-signals-default-000001] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [.siem-signals-default]
[00:00:19]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-ubuntu-18-tests-xl-1585688074551984762] moving index [.siem-signals-default-000001] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] in policy [.siem-signals-default]
[00:00:19]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-ubuntu-18-tests-xl-1585688074551984762] moving index [.siem-signals-default-000001] from [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-follow-shard-tasks"}] in policy [.siem-signals-default]
[00:00:24]                 │ info [o.e.x.s.a.AuthenticationService] [kibana-ci-immutable-ubuntu-18-tests-xl-1585688074551984762] Authentication using apikey failed - api key has been invalidated
[00:00:24]                 │ proc [kibana]   log   [22:05:56.589] [error][plugins][siem] An error occurred during rule execution:
[00:00:24]                 │ proc [kibana] message: "[security_exception] missing authentication credentials for REST request [/apm-*-transaction*%2Cauditbeat-*%2Cendgame-*%2Cfilebeat-*%2Cpacketbeat-*%2Cwinlogbeat-*/_search?allow_no_indices=true&size=100&ignore_unavailable=true], with { header={ WWW-Authenticate={ 0="Bearer realm=\"security\"" & 1="ApiKey" & 2="Basic realm=\"security\" charset=\"UTF-8\"" } } }"
[00:00:24]                 │ proc [kibana] name: "Simple Rule Query"
[00:00:24]                 │ proc [kibana] id: "df7f8331-b28d-4af6-9f8a-303abf2218fc"
[00:00:24]                 │ proc [kibana] rule id: "rule-1"
[00:00:24]                 │ proc [kibana] signals index: ".siem-signals-default"
[00:00:24]                 │ info [o.e.x.s.a.AuthenticationService] [kibana-ci-immutable-ubuntu-18-tests-xl-1585688074551984762] Authentication using apikey failed - api key has been invalidated
[00:00:24]                 │ proc [kibana]   log   [22:05:56.605] [error][alerting][alerting][plugins][plugins] Executing Alert "df7f8331-b28d-4af6-9f8a-303abf2218fc" has resulted in Error: [security_exception] missing authentication credentials for REST request [/_security/user/_has_privileges], with { header={ WWW-Authenticate={ 0="Bearer realm=\"security\"" & 1="ApiKey" & 2="Basic realm=\"security\" charset=\"UTF-8\"" } } }
[00:00:24]                 └- ✓ pass  (5.0s) "detection engine api security and spaces enabled create_rules creating rules should create a single rule without an input index"
[00:00:24]               └-> "after each" hook
[00:00:24]                 │ info [o.e.c.m.MetaDataDeleteIndexService] [kibana-ci-immutable-ubuntu-18-tests-xl-1585688074551984762] [.siem-signals-default-000001/pyG8UyAFTGKkM8smDLX47A] deleting index
[00:00:25]                 │ info [o.e.c.m.MetaDataIndexTemplateService] [kibana-ci-immutable-ubuntu-18-tests-xl-1585688074551984762] removing template [.siem-signals-default]
[00:00:25]               └-> should create a single rule without a rule_id
[00:00:25]                 └-> "before each" hook: global before each
[00:00:25]                 └-> "before each" hook
[00:00:25]                   │ info [o.e.x.i.a.TransportPutLifecycleAction] [kibana-ci-immutable-ubuntu-18-tests-xl-1585688074551984762] adding index lifecycle policy [.siem-signals-default]
[00:00:25]                   │ info [o.e.c.m.MetaDataIndexTemplateService] [kibana-ci-immutable-ubuntu-18-tests-xl-1585688074551984762] adding template [.siem-signals-default] for index patterns [.siem-signals-default-*]
[00:00:25]                   │ info [o.e.c.m.MetaDataCreateIndexService] [kibana-ci-immutable-ubuntu-18-tests-xl-1585688074551984762] [.siem-signals-default-000001] creating index, cause [api], templates [.siem-signals-default], shards [1]/[1], mappings [_doc]
[00:00:25]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-ubuntu-18-tests-xl-1585688074551984762] moving index [.siem-signals-default-000001] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [.siem-signals-default]
[00:00:25]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-ubuntu-18-tests-xl-1585688074551984762] moving index [.siem-signals-default-000001] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] in policy [.siem-signals-default]
[00:00:25]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-ubuntu-18-tests-xl-1585688074551984762] moving index [.siem-signals-default-000001] from [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-follow-shard-tasks"}] in policy [.siem-signals-default]
[00:00:29]                 └- ✓ pass  (4.4s) "detection engine api security and spaces enabled create_rules creating rules should create a single rule without a rule_id"
[00:00:29]               └-> "after each" hook
[00:00:29]                 │ info [o.e.c.m.MetaDataDeleteIndexService] [kibana-ci-immutable-ubuntu-18-tests-xl-1585688074551984762] [.siem-signals-default-000001/ylCjztfBR3aqZF7uPCfdwQ] deleting index
[00:00:30]                 │ info [o.e.c.m.MetaDataIndexTemplateService] [kibana-ci-immutable-ubuntu-18-tests-xl-1585688074551984762] removing template [.siem-signals-default]
[00:00:30]               └-> should create a single Machine Learning rule
[00:00:30]                 └-> "before each" hook: global before each
[00:00:30]                 └-> "before each" hook
[00:00:30]                   │ info [o.e.x.i.a.TransportPutLifecycleAction] [kibana-ci-immutable-ubuntu-18-tests-xl-1585688074551984762] adding index lifecycle policy [.siem-signals-default]
[00:00:30]                   │ info [o.e.c.m.MetaDataIndexTemplateService] [kibana-ci-immutable-ubuntu-18-tests-xl-1585688074551984762] adding template [.siem-signals-default] for index patterns [.siem-signals-default-*]
[00:00:30]                   │ info [o.e.c.m.MetaDataCreateIndexService] [kibana-ci-immutable-ubuntu-18-tests-xl-1585688074551984762] [.siem-signals-default-000001] creating index, cause [api], templates [.siem-signals-default], shards [1]/[1], mappings [_doc]
[00:00:30]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-ubuntu-18-tests-xl-1585688074551984762] moving index [.siem-signals-default-000001] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [.siem-signals-default]
[00:00:30]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-ubuntu-18-tests-xl-1585688074551984762] moving index [.siem-signals-default-000001] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] in policy [.siem-signals-default]
[00:00:30]                 │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-ubuntu-18-tests-xl-1585688074551984762] moving index [.siem-signals-default-000001] from [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-follow-shard-tasks"}] in policy [.siem-signals-default]
[00:00:33]                 │ proc [kibana]   log   [22:06:05.834] [warning][plugins][siem] Machine learning job is not started:
[00:00:33]                 │ proc [kibana] job id: "some_job_id"
[00:00:33]                 │ proc [kibana] job status: "undefined"
[00:00:33]                 │ proc [kibana] datafeed status: "undefined"
[00:00:33]                 │ proc [kibana] name: "Simple ML Rule"
[00:00:33]                 │ proc [kibana] id: "6db94cc1-7b5b-4e3f-b318-a87e57f05384"
[00:00:33]                 │ proc [kibana] rule id: "rule-1"
[00:00:33]                 │ proc [kibana] signals index: ".siem-signals-default"
[00:00:35]                 └- ✖ fail: "detection engine api security and spaces enabled create_rules creating rules should create a single Machine Learning rule"
[00:00:35]                 │

Stack Trace

{ Error: expected { created_by: 'elastic',
  description: 'Simple Machine Learning Rule',
  enabled: true,
  false_positives: [],
  from: 'now-6m',
  immutable: false,
  interval: '5m',
  rule_id: 'rule-1',
  output_index: '.siem-signals-default',
  max_signals: 100,
  risk_score: 1,
  name: 'Simple ML Rule',
  references: [],
  severity: 'high',
  updated_by: 'elastic',
  tags: [],
  to: 'now',
  type: 'machine_learning',
  threat: [],
  version: 1,
  lists: [],
  actions: [],
  throttle: 'no_actions',
  last_failure_at: '2020-03-31T22:06:05.853Z',
  last_failure_message: 'Machine learning job is not started:\njob id: "some_job_id"\njob status: "undefined"\ndatafeed status: "undefined"\nname: "Simple ML Rule"\nid: "6db94cc1-7b5b-4e3f-b318-a87e57f05384"\nrule id: "rule-1"\nsignals index: ".siem-signals-default"',
  anomaly_threshold: 44,
  machine_learning_job_id: 'some_job_id' } to sort of equal { actions: [],
  created_by: 'elastic',
  description: 'Simple Machine Learning Rule',
  enabled: true,
  false_positives: [],
  from: 'now-6m',
  immutable: false,
  interval: '5m',
  rule_id: 'rule-1',
  output_index: '.siem-signals-default',
  max_signals: 100,
  risk_score: 1,
  name: 'Simple ML Rule',
  references: [],
  severity: 'high',
  updated_by: 'elastic',
  tags: [],
  to: 'now',
  type: 'machine_learning',
  threat: [],
  throttle: 'no_actions',
  lists: [],
  version: 1,
  anomaly_threshold: 44,
  machine_learning_job_id: 'some_job_id' }
    at Assertion.assert (/dev/shm/workspace/kibana/packages/kbn-expect/expect.js:100:11)
    at Assertion.eql (/dev/shm/workspace/kibana/packages/kbn-expect/expect.js:244:8)
    at Context.it (test/detection_engine_api_integration/security_and_spaces/tests/create_rules.ts:101:34)
  actual:
   '{\n  "actions": []\n  "anomaly_threshold": 44\n  "created_by": "elastic"\n  "description": "Simple Machine Learning Rule"\n  "enabled": true\n  "false_positives": []\n  "from": "now-6m"\n  "immutable": false\n  "interval": "5m"\n  "last_failure_at": "2020-03-31T22:06:05.853Z"\n  "last_failure_message": "Machine learning job is not started:\\njob id: \\"some_job_id\\"\\njob status: \\"undefined\\"\\ndatafeed status: \\"undefined\\"\\nname: \\"Simple ML Rule\\"\\nid: \\"6db94cc1-7b5b-4e3f-b318-a87e57f05384\\"\\nrule id: \\"rule-1\\"\\nsignals index: \\".siem-signals-default\\""\n  "lists": []\n  "machine_learning_job_id": "some_job_id"\n  "max_signals": 100\n  "name": "Simple ML Rule"\n  "output_index": ".siem-signals-default"\n  "references": []\n  "risk_score": 1\n  "rule_id": "rule-1"\n  "severity": "high"\n  "tags": []\n  "threat": []\n  "throttle": "no_actions"\n  "to": "now"\n  "type": "machine_learning"\n  "updated_by": "elastic"\n  "version": 1\n}',
  expected:
   '{\n  "actions": []\n  "anomaly_threshold": 44\n  "created_by": "elastic"\n  "description": "Simple Machine Learning Rule"\n  "enabled": true\n  "false_positives": []\n  "from": "now-6m"\n  "immutable": false\n  "interval": "5m"\n  "lists": []\n  "machine_learning_job_id": "some_job_id"\n  "max_signals": 100\n  "name": "Simple ML Rule"\n  "output_index": ".siem-signals-default"\n  "references": []\n  "risk_score": 1\n  "rule_id": "rule-1"\n  "severity": "high"\n  "tags": []\n  "threat": []\n  "throttle": "no_actions"\n  "to": "now"\n  "type": "machine_learning"\n  "updated_by": "elastic"\n  "version": 1\n}',
  showDiff: true }

---

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream