[Draft] Followup issues for alerting saved object sharing

[ymao1]

No description provided.

[ymao1]

[Maps][Alerting] Migrate to using Saved Object References array to reference index patterns from Rules

Currently, geo containment rules stores two index pattern Saved Object IDs inside the rule params inside params.indexId and params.boundaryIndexId:

Example geo containment rule

{
	"alert": {
		"params": {
			"index": "tracks*",
			"indexId": "f3acd150-e8c6-11eb-bc58-13b6ae6d011a",
			"entity": "azimuth",
			"dateField": "@timestamp",
			"boundaryType": "entireIndex",
			"geoField": "location",
			"boundaryIndexTitle": "manhattan_boundaries",
			"boundaryIndexId": "a2e7a1f0-e8c6-11eb-bc58-13b6ae6d011a",
			"boundaryGeoField": "coordinates",
			"boundaryNameField": "<nothing selected>"
		},
		"consumer": "alerts",
		"schedule": {
			"interval": "1m"
		},
		"tags": [],
		"name": "map",
		"actions": [],
		"enabled": true,
		"throttle": null,
		"alertTypeId": ".geo-containment",
		"notifyWhen": "onActionGroupChange",
		"apiKeyOwner": "elastic",
		"apiKey": "HcRdCvA7pjzG170nbE5NJdtSPB2fQ/yKy3i+rOApxnjCw/G/7MaA5X1sSblceYMD5u7eoot3CKiFFOgKeiz3w9Glmvnk4scpGLAtHdQYn4lUtY5m5lYaO5Xzj+v8SIN/Ogqpx8/Tg+qLSVPRlpEWNrBh+sPnL0k0SofOfUkIurU4TRJQq4yDjZIf+DI6hgA1qOFd1aQckgnq0A==",
		"createdBy": "elastic",
		"updatedBy": "elastic",
		"createdAt": "2021-07-19T19:25:15.787Z",
		"updatedAt": "2021-07-19T19:25:15.787Z",
		"muteAll": false,
		"mutedInstanceIds": [],
		"executionStatus": {
			"status": "active",
			"lastExecutionDate": "2021-07-19T19:26:22.339Z",
			"error": null
		},
		"meta": {
			"versionApiKeyLastmodified": "8.0.0"
		},
		"scheduledTaskId": "0c1068b0-e8c7-11eb-bc58-13b6ae6d011a"
	},
	"type": "alert",
	"references": [],
	"namespace": "my-space",
	"migrationVersion": {
		"alert": "7.13.0"
	},
	"coreMigrationVersion": "8.0.0",
	"updated_at": "2021-07-19T19:26:22.631Z"
}

In 8.0, saved objects will be converted to share-capable (meta issue). This means all index pattern SO IDs will be regenerated and the index pattern SO IDs stored within the geo containment rule params will be outdated.

What does this mean?

Rule Execution
It doesn't look like the SO IDs are actually directly used during geo containment rule execution so rule execution continues to run successfully even if the index pattern IDs are outdated.

Rules UI
Index pattern SO IDs are used to re-populate the UI field when editing the geo containment rule so because the index pattern SO IDs are outdated, the edit form was cleared out.

Recommended approach

With this resolution of this issue with this PR, the alerting framework allows rule types to specify extract and inject hooks for pulling out SO references from rule parameters. POC for adding it to the geo containment rule type is available here: https://github.com/elastic/kibana/pull/106733/files. While these hooks will affect rules that are created/updated after implementation, a migration will also need to be added for the geo containment rule to update existing rules.

Alternative approach

Since the indexId and boundaryIndexId SO IDs are only used during rule editing, there is the option to use the client side .resolve function provided by the saved objects client in order to resolve the outdated index pattern SO ID.

When does this need to be done?

[ymao1]

[Cases][Connectors] Migrate to using Saved Object References array to reference action saved objects from Cases

Currently, there are several cases saved object types that directly store the associated actions SO ID instead of utilizing the references array. With alerting planning to make actions share-capable in 8.0.0, the SO IDs for these actions will be re-generated, which will break the association in these cases saved objects.

Cases SO

Stores the associated action SO ID in the cases.connector.id field

Example `cases` saved object

{
	"_index": ".kibana_8.0.0_001",
	"_id": "cases:554e2720-e8b2-11eb-bc58-13b6ae6d011a",
	"_score": 0.0,
	"_source": {
		"cases": {
			"type": "individual",
			"title": "test case",
			"tags": [],
			"description": "hi",
			"connector": {
				"id": "7b0cf000-e8a7-11eb-bc58-13b6ae6d011a", <---- action SO ID
				"name": "test servicenow",
				"type": ".servicenow",
				"fields": [{
						"key": "urgency",
						"value": null
					},
					{
						"key": "severity",
						"value": null
					},
					{
						"key": "impact",
						"value": null
					},
					{
						"key": "category",
						"value": null
					},
					{
						"key": "subcategory",
						"value": null
					}
				]
			},
			"settings": {
				"syncAlerts": true
			},
			"owner": "securitySolution",
			"closed_at": null,
			"closed_by": null,
			"created_at": "2021-07-19T16:56:59.796Z",
			"created_by": {
				"email": null,
				"full_name": null,
				"username": "elastic"
			},
			"external_service": null,
			"status": "open",
			"updated_at": "2021-07-19T17:04:19.545Z",
			"updated_by": {
				"full_name": null,
				"email": null,
				"username": "elastic"
			}
		},
		"type": "cases",
		"references": [],
		"migrationVersion": {
			"cases": "7.14.0"
		},
		"coreMigrationVersion": "8.0.0",
		"updated_at": "2021-07-19T17:04:19.548Z"
	}
}

When the action SO is made share-capable, this error will show in the Cases UI because the ID has changed.

Screenshot of error

Cases User Actions SO

Stores the associated action SO ID when the user changes the connector associated with a case. The old and new action SO IDs are stored as part of a stringified JSON.

Example `cases-user-actions` saved object

{
	"_index": ".kibana_8.0.0_001",
	"_id": "cases-user-actions:99836640-e8a7-11eb-bc58-13b6ae6d011a",
	"_score": 0.0,
	"_source": {
		"cases-user-actions": {
			"action_field": [
				"connector"
			],
			"action": "update",
			"action_at": "2021-07-19T15:40:09.374Z",
			"action_by": {
				"email": null,
				"full_name": null,
				"username": "elastic"
			},
			"new_value": """{"id ":"7b0cf000- e8a7- 11eb-bc58-13b6ae6d011a ","name ":"test servicenow ","type ":".servicenow ","fields ":null}""",
			"old_value": """{"id ":"6e67ec46-28af-4c19-8f19-683d6223ae26","name ":"service now personal developer instance ","type ":".servicenow ","fields ":{"urgency ":null,"severity ":null,"impact ":null,"category ":null,"subcategory ":null}}""",
			"owner": "securitySolution"
		},
		"type": "cases-user-actions",
		"references": [{
			"type": "cases",
			"name": "associated-cases",
			"id": "270e4080-e8a7-11eb-bc58-13b6ae6d011a"
		}],
		"migrationVersion": {
			"cases-user-actions": "7.14.0"
		},
		"coreMigrationVersion": "8.0.0",
		"updated_at": "2021-07-19T15:40:09.764Z"
	}
}

When the action SO is made share-capable, the Cases UI will display a user action message of ${user} removed external incident management system instead of the correct user action.

Cases Configure SO

Stores the associated action SO ID in the connector.id field.

Example `cases-configure` saved object

{
	"_index": ".kibana_8.0.0_001",
	"_id": "cases-configure:2d5dbcc0-e8b3-11eb-bc58-13b6ae6d011a",
	"_score": 0.0,
	"_source": {
		"cases-configure": {
			"connector": {
				"id": "7b0cf000-e8a7-11eb-bc58-13b6ae6d011a",
				"name": "test servicenow",
				"type": ".servicenow",
				"fields": []
			},
			"closure_type": "close-by-user",
			"owner": "securitySolution",
			"created_at": "2021-07-19T17:03:02.286Z",
			"created_by": {
				"username": "elastic",
				"email": null,
				"full_name": null
			},
			"updated_at": null,
			"updated_by": null
		},
		"type": "cases-configure",
		"references": [],
		"migrationVersion": {
			"cases-configure": "7.14.0"
		},
		"coreMigrationVersion": "8.0.0",
		"updated_at": "2021-07-19T17:03:02.571Z"
	}
}

When the action SO is made share-capable, the Configure Cases UI will incorrect state that the associated connector has been deleted.

Screenshot of message

Note that the cases-connector-mappings SO correctly uses the references array to store the associated action SO.

Recommended approach

The recommended approach is to update the cases, cases-user-actions and cases-configure saved objects so that associated action SO IDs are stored inside the references array. This requires a code update to extracted the associated action SO ID on create/update from the document into the references array and to inject the ID on find/get. This would also require a migration to update existing SOs of this type to use the references array. Once this is done, the saved object service will takes care of updating the references when action SOs are made share-capable.

Alternative approach

Alternatively, the cases team could explore using the .resolve() functionality provided by the saved object service in order to resolve the outdated action SO ID to their re-generated ID. This would require tracking down all the places where the cases, cases-user-actions and cases-configure saved objects are used to retrieve the associated actiohn SO and injecting the resolve layer to ensure the correct SO is loaded. This seems like a more brittle approach.

When does this need to be done?

[ymao1]

Existing security issue: #105819, #101975

[Security Rules] Migrate to using Saved Object References array to reference alert saved objects from security saved objects

Currently, there are several security saved object types that directly store the associated rule (alert) SO ID instead of utilizing the references array. With alerting planning to make alerts share-capable in 8.0.0, the SO IDs for these rules will be re-generated, which will break the association in these security saved objects.

Detection Engine Rule Status SO

Stores the associated alert SO ID in the alertId field

Example `siem-detection-engine-rule-status` saved object

{
	"_index": ".kibana_8.0.0_001",
	"_id": "siem-detection-engine-rule-status:ab8f4fc0-e8b6-11eb-bc58-13b6ae6d011a",
	"_score": 0.0,
	"_source": {
		"siem-detection-engine-rule-status": {
			"alertId": "a9f99490-e8b6-11eb-bc58-13b6ae6d011a",
			"statusDate": "2021-07-19T17:34:38.755Z",
			"status": "partial failure",
			"lastFailureAt": null,
			"lastSuccessAt": "2021-07-19T17:34:38.755Z",
			"lastFailureMessage": null,
			"lastSuccessMessage": """The following indices are missing the timestamp field "@timestamp ": [".kibana_8 .0 .0_001 "]""",
			"gap": null,
			"bulkCreateTimeDurations": [],
			"searchAfterTimeDurations": [],
			"lastLookBackDate": null
		},
		"type": "siem-detection-engine-rule-status",
		"references": [],
		"coreMigrationVersion": "8.0.0",
		"updated_at": "2021-07-19T17:34:38.757Z"
	}
}

When the alert SO is made share-capable, no statuses will be returned for the rules in the detection UI. However, maybe this is ok? There doesn't seem to be any errors in the UI and since I think only the last few statuses are shown, user would just have to wait for the rule to run again with the new ID to start seeing statuses.

Detection Engine Rule Action SO

Stores the associated alert SO ID in the ruleAlertId field

Example `siem-detection-engine-rule-action` saved object

{
	"_index": ".kibana_8.0.0_001",
	"_id": "siem-detection-engine-rule-actions:ab321490-e8b6-11eb-bc58-13b6ae6d011a",
	"_score": 0.0,
	"_source": {
		"siem-detection-engine-rule-actions": {
			"ruleAlertId": "a9f99490-e8b6-11eb-bc58-13b6ae6d011a",
			"actions": [{
				"action_type_id": ".email",
				"id": "8fd8e910-e8a4-11eb-bc58-13b6ae6d011a",
				"params": {
					"subject": "hi",
					"to": [
						"me@me.com"
					],
					"message": "Rule {{context.rule.name}} generated {{state.signals_count}} alerts"
				},
				"group": "default"
			}],
			"ruleThrottle": "rule",
			"alertThrottle": null
		},
		"type": "siem-detection-engine-rule-actions",
		"references": [],
		"migrationVersion": {
			"siem-detection-engine-rule-actions": "7.11.2"
		},
		"coreMigrationVersion": "8.0.0",
		"updated_at": "2021-07-19T17:37:28.027Z"
	}
}

When the alert SO is made share-capable, the association between siem-detection-engine-rule-actions SO and its detection rule is lost, so it looks like no notification actions would be scheduled anymore for detection rules. When retrieving the rule through the security UI or API, the actions array always returns as empty.

Recommended approach

The recommended approach is to update the siem-detection-engine-rule-status and siem-detection-engine-rule-actions saved objects to use the references array. This requires a code update to extracted the associated alert SO ID on create/update from the document into the references array and to inject the ID on find/get. This would also require a migration to update existing SOs of this type to use the references array. Once this is done, the saved object service will takes care of updating the references when alert SOs are made share-capable.

Alternative approach

Alternatively, the security team could explore using the .resolve() functionality provided by the saved object service in order to resolve the outdated alert SO ID to the re-generated ID. This would require tracking down all the places where the siem-detection-engine-rule-status and siem-detection-engine-rule-actions saved objects are used to retrieve the associated alert SO and injecting the resolve layer to ensure the correct SO is loaded. This seems like a more brittle approach.

When does this need to be done?