Request for alerting internal tags structure

[FrankHassanabad]

Describe the feature:
Add internal tags structure for consumers of the Alert API that are not use facing. Right now we are using on SIEM internal structures and trying to keep them filtered from the front end. It would be better if we separate the two tags from each other of internal and public facing tags in two different structures.

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[pmuellr]

I think this is a case where we would want the "system" tags (vs "user" tags) to not be settable via the HTTP APIs, only via the alerting client. I think it probably makes sense to make them visible to users though (get requests and searches).

[FrankHassanabad]

It's ok to have them visible but not changeable and to have system tags. This is a good approach that I agree with 👍

[sorenlouv]

+1 for system tags. This is super important for being able to reliably query for alerts based on solution specific identifiers.
I don't think it's terrible important that system tags are protected on the API level - but having the distinction between system tags and user tags will make it possible to at least not show them in the UI which makes it way to easy for end users to inadvertently break things.

[FrankHassanabad]

"Reviewed by Frank Hassanabad on 7/29/2020, still valid as of this date"

[alexfrancoeur]

User generated tags for saved objects (#74571) is currently in development. While these aren't internal or system tags, I wonder if it's something our alerting framework can leverage. Tags and a combination of "read only assets" (#70461) may end up meeting most of the requirements here.

[sorenlouv]

Edit: this might already have been discussed in the linked issue. Will read through that now.

---

@alexfrancoeur Will these tags support key/value lookups, eg. "service.environment": "production" ?

It is important that it's not simply a single value lookup like: production since the same value could be used for different fields (eg. "demo" could both be a service environment and a service name).
We are running into this problem right now and it makes it much harder impossible to incorporate alerts in the UIs.

One approach we've taken is to use a reserved keyword (eg. :) as separator: "service.environment:production". This however breaks down if the reserved character is used in the environment.

[mikecote]

cc @arisonl

[mikecote]

Moving from 7.12 - Candidates to 7.x - Candidates.

[mikecote]

Moving from 7.x - Candidates to 8.x - Candidates (Backlog) after the latest 7.x planning session.

[mikecote]

I just merged the capability of filtering alerts by string parameters (#92036). This should remove the need to use internal tags altogether as you can find alerts based on keyword values as long as they are parameters to the alert. I can't think of any extra information you don't want to become a parameter to the alert executor so I will close the issue Wednesday, Feb 24th if there are no problems/use cases.

[spong]

Thanks @mikecote! I've opened #92334 for refactoring our internal_tags workaround.

Testing real quick though, I modified:

kibana/x-pack/plugins/security_solution/public/detections/containers/detection_engine/rules/api.ts

Lines 113 to 118 in 4584a8b

	const showCustomRuleFilter = filterOptions.showCustomRules
	? [`alert.attributes.tags: "__internal_immutable:false"`]
	: [];
	const showElasticRuleFilter = filterOptions.showElasticRules
	? [`alert.attributes.tags: "__internal_immutable:true"`]
	: [];

to be:

  const showCustomRuleFilter = filterOptions.showCustomRules
    ? [`alert.attributes.params.immutable: "false"`]
    : [];
  const showElasticRuleFilter = filterOptions.showElasticRules
    ? [`alert.attributes.params.immutable: "true"`]
    : [];

And this appears to work! 🎉 So I think this should be suitable for us to remove this chunk of tech debt barring any other unforeseen issues. Thank you for your efforts here! 🙂

[pmuellr]

@sqren Do the new flattened params work for you as well? I think we can close this issue, if so.

[sorenlouv]

I have yet to try it out but yes, I agree that it removes the need for internal tags - at least for my use cases.

[sorenlouv]

Btw. this is super awesome. Thanks for working so hard on it and finally getting it in!

[mikecote]

I will consider this issue now resolved and closed. Thanks all!