[SIEM][Detection Engine][Lists] Adds conflict versioning and io-ts improvements to lists

[FrankHassanabad]

Summary

• Adds conflict versioning by exposing the "_version" from the saved object system. It renames "version" to "_version" so that we can use regular "version" later for versioning things for pre-packaged lists abilities.
• Utilizes t.OutputOf in the requests and the data types to give us more correctly types
• Removes the Identity utility as that is adding confusion and can confuse vs code rather than improves things
• Removes extra types that were causing confusion which was an idiom from io-ts
• Changes the wording of Partial by removing that and instead focuses the request types on either client side or server side at this point.

NOTE: The UI can migrate to holding onto the _version and then push it back down when it wants to migrate to using the conflict resolution. If the UI does not push it down, then a value of undefined will be used which is indicating that no conflict errors are wanted.

Output example of posting an exception list:

❯ ./post_exception_list.sh

{
  "_tags": [
    "endpoint",
    "process",
    "malware",
    "os:linux"
  ],
  "_version": "Wzk4NiwxXQ==",
  "created_at": "2020-07-17T18:59:22.872Z",
  "created_by": "yo",
  "description": "This is a sample endpoint type exception",
  "id": "a08795b0-c85f-11ea-b1a6-c155df988a92",
  "list_id": "simple_list",
  "name": "Sample Endpoint Exception List",
  "namespace_type": "single",
  "tags": [
    "user added string for a tag",
    "malware"
  ],
  "tie_breaker_id": "b789ec05-3e0f-4344-a156-0c0f5b6e2f9c",
  "type": "detection",
  "updated_at": "2020-07-17T18:59:22.891Z",
  "updated_by": "yo"
}

Output example of posting an exception list item
❯ ./post_exception_list_item.sh

{
  "_tags": [
    "endpoint",
    "process",
    "malware",
    "os:linux"
  ],
  "_version": "Wzk4NywxXQ==",
  "comments": [],
  "created_at": "2020-07-17T18:59:30.286Z",
  "created_by": "yo",
  "description": "This is a sample endpoint type exception",
  "entries": [
    {
      "field": "actingProcess.file.signer",
      "operator": "excluded",
      "type": "exists"
    },
    {
      "field": "host.name",
      "operator": "included",
      "type": "match_any",
      "value": [
        "some host",
        "another host"
      ]
    }
  ],
  "id": "a4f2b800-c85f-11ea-b1a6-c155df988a92",
  "item_id": "simple_list_item",
  "list_id": "simple_list",
  "name": "Sample Endpoint Exception List",
  "namespace_type": "single",
  "tags": [
    "user added string for a tag",
    "malware"
  ],
  "tie_breaker_id": "1dc456bc-7aa9-44b4-bca3-131689cf729f",
  "type": "simple",
  "updated_at": "2020-07-17T18:59:30.304Z",
  "updated_by": "yo"
}

Output example of when you get an exception list:

❯ ./get_exception_list.sh simple_list

{
  "_tags": [
    "endpoint",
    "process",
    "malware",
    "os:linux"
  ],
  "_version": "WzEwNzcsMV0=",
  "created_at": "2020-07-17T18:59:22.872Z",
  "created_by": "yo",
  "description": "Different description",
  "id": "a08795b0-c85f-11ea-b1a6-c155df988a92",
  "list_id": "simple_list",
  "name": "Sample Endpoint Exception List",
  "namespace_type": "single",
  "tags": [
    "user added string for a tag",
    "malware"
  ],
  "tie_breaker_id": "b789ec05-3e0f-4344-a156-0c0f5b6e2f9c",
  "type": "endpoint",
  "updated_at": "2020-07-17T20:01:24.958Z",
  "updated_by": "yo"
}

Example of the error you get if you do an update of an exception list and someone else has changed it:

{
  "message": "[exception-list:a08795b0-c85f-11ea-b1a6-c155df988a92]: version conflict, required seqNo [1074], primary term [1]. current document has seqNo [1077] and primary term [1]: [version_conflict_engine_exception] [exception-list:a08795b0-c85f-11ea-b1a6-c155df988a92]: version conflict, required seqNo [1074], primary term [1]. current document has seqNo [1077] and primary term [1], with { index_uuid=\"a2mgXBO6Tl2ULDq-MTs1Tw\" & shard=\"0\" & index=\".kibana-hassanabad_1\" }",
  "status_code": 409
}

Lists are the same way and flavor, they encode the _version the same way that saved objects do. To see those work you run these scripts:

./post_list.sh
./post_list_item.sh
./find_list.sh
./find_list_item.sh

Checklist

• Unit or functional tests were updated or added to match the most common scenarios

[elasticmachine]

Pinging @elastic/siem (Team:SIEM)

[madirey]

@FrankHassanabad Pulled this down, ran through several tests using the scripts. Updates to lists, list items, exception lists, and exception list items work with and without _version. Expected 409 is returned when a version is re-used. Looks good!

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: 191cf26

Build metrics

page load bundle size

id	value	diff	baseline
lists	256.7KB	+503.0B	256.2KB

History

• 💚 Build #62793 succeeded ab972a6
• 💔 Build #62748 failed adb5ba5

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)