[Auditbeat] event.dataset should be "module.metricset"

[andrewkroh]

To align with both Filebeat and Metricbeat the event.dataset field should be populated by Auditbeat with {module}.{metricset}. Currently the value is populated with only the metricset name (i.e. event.dataset: "socket").

This only affects the system module in Auditbeat.

[andrewkroh]

@ruflin @webmat Do you agree? The system module is still beta so we could make this change now to get alignment.

[ruflin]

@andrewkroh Yes! And the event.dataset value should be unique and not overlap with metricbeat (if the data is not the same). But I assume that is already the case? I'm surprised the above is not already the case, because Metricbeat should do the above by default.

[webmat]

Yes, I totally agree!

It's even a clarification I'm in the process of adding to ECS. You can see it here.

I'm not making this a hard requirement over there, as I have in mind other implementers as well. But I think it would be best if we strived to do that as much as possible within the Elastic Stack.

[andrewkroh]

@ruflin Auditbeat has some slightly different handling from Metricbeat because the auditd and file_integrity modules don't have "datasets".

beats/auditbeat/core/eventmod.go

Lines 34 to 35 in a5bdf52

	// Modules without "datasets" should set their module and metricset names
	// to the same value then this will omit the event.dataset field.

[ruflin]

Got it, I slightly remember a discussion around this.

I'm thinking if we should rename event.Namespace to event.Namespace because now that we switched everything over to the new reporter, that is what it is used for mostly: https://github.com/elastic/beats/blob/master/metricbeat/mb/event.go#L124 Or we should introduce event.Dataset and move it over gradually.

@exekias FYI

[urso]

@exekias @andrewkroh Is this issue resolved by now, or would it be a breaking change and we should postpone this to 8.0?

[consulthys]

@ruflin any insights about this one and an idea if/when the change will make it? Thank you so much!

[ruflin]

@consulthys What part are you worried about, the internals of MB or what the end event might look like? One thing that changed since we discussed this is that we introduced a new indexing strategy which might give a new perspective on this issue.

[consulthys]

@ruflin yes, that's exactly the new indexing strategy (%{type}-%{event.dataset}-%{namespace}) that got me thinking. My question is how does Auditbeat approach the new indexing strategy? Will event.dataset be filled in?

[ruflin]

@consulthys The fields used for the new indexing strategy are data_stream.*: elastic/ecs#1145 These are added by the Elastic Agent when running a Beat. Today we do not support Auditbeat yet but if we add it, these 3 fields must be filled in.

[consulthys]

@ruflin thank you very much for mentioning which fields are actually used to create the index name 👍
But can you also give a quick insight on which part of the pipeline will actually be using those three fields to create the final index name? Is it a processor on Agent/Beats side or an ingest pipeline on ES side?

[ruflin]

Currently it the data stream selection is done by the Elastic Agent. Elasticsearch will use the fields to set the correct mappings.

Here is also a first blog post that describes these fields a bit more in detail: https://www.elastic.co/blog/an-introduction-to-the-elastic-data-stream-naming-scheme

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[botelastic]

This issue doesn't have a Team:<team> label.

[mtojek]

Stale, resolving