Locality values mismatch between Netflow input and ingest pipeline

[adriansr]

For confirmed bugs, please report:

• Version: 7.11
• Operating System: -
• Discuss Forum URL: -
• Steps to Reproduce: -

Netflow input will populate {source,destination,flow}.locality fields with values either private or public, depending on the internal_networks configuration option.

The ingest pipeline for the netflow module uses those fields to populate network.direction. However, it is expecting the values to be internal or external, which causes network.direction to always be unknown.

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[adriansr]

To clarify the issue:

The Netflow input will set locality fields to one of two possible values: private or public.

beats/x-pack/filebeat/input/netflow/convert.go

Lines 385 to 388 in 6986c84

	var localityNames = map[Locality]string{
	LocalityPrivate: "private",
	LocalityPublic: "public",
	}

The Netflow module ingest pipeline uses those values to set network.direction, but it expects either internal or external:

beats/x-pack/filebeat/module/netflow/log/ingest/pipeline.yml

Lines 54 to 73 in 6986c84

	- set:
	field: network.direction
	value: inbound
	if: 'ctx?.source?.locality == "external" && ctx?.destination?.locality == "internal"'
	- set:
	field: network.direction
	value: outbound
	if: 'ctx?.source?.locality == "internal" && ctx?.destination?.locality == "external"'
	- set:
	field: network.direction
	value: internal
	if: 'ctx?.source?.locality == "internal" && ctx?.destination?.locality == "internal"'
	- set:
	field: network.direction
	value: external
	if: 'ctx?.source?.locality == "external" && ctx?.destination?.locality == "external"'
	- set:
	field: network.direction
	value: unknown
	if: 'ctx?.network?.direction == null'

We need to choose which pair of values to use, private/public or internal/external.

[andrewkroh]

In addition to the mismatch of values, there is another bug that could be at play #24110 since this would cause there to be no default value for internal_networks CIDRs.