[Bug] Handle formatting empty list

[Mikaayenson]

Pull Request

Issue link(s):
Related to #4066
Related to https://elasticstack.slack.com/archives/C06TE19EP09/p1726588617531219?thread_ts=1726587438.668979&cid=C06TE19EP09

Summary - What I changed

• This PR introduced a bug where formatting adjusted the fields for empty lists, which may be the case when exporting rules using the DAC features.

How To Test

Running python -m detection_rules kibana --space main export-rules -d custom_rules for a custom rule to see the broken output.

Sample Rule After Export without Fix

[metadata]
creation_date = "2024/09/17"
integration = 
  [

  ]

maturity = "production"
updated_date = "2024/09/17"

[rule]
actions = 
  [

  ]

author = ["513814043"]
description = "Test ESQL Suppress Rule"
enabled = false
exceptions_list = 
  [

  ]

false_positives = 
  [

  ]

from = "now-360s"
interval = "5m"
language = "esql"
license = ""
max_signals = 100
name = "Test ESQL Suppress Rule"
references = 
  [

  ]

related_integrations = 
  [

  ]

required_fields = 
  [

  ]

revision = 0
risk_score = 21
risk_score_mapping = 
  [

  ]

rule_id = "058a8221-5b41-49ad-9e68-5a60fdf977e8"
setup = ""
severity = "low"
severity_mapping = 
  [

  ]

tags = 
  [

  ]

threat = 
  [

  ]

to = "now"
type = "esql"
version = 1

query = '''
from logs* metadata _id, _version, _index |
    WHERE process.name == "test"
'''



[rule.alert_suppression]
group_by = ["process.name"]
missing_fields_strategy = "suppress"

Sample Rule After Export w/Fix

[metadata]
creation_date = "2024/09/17"
integration = []
maturity = "production"
updated_date = "2024/09/17"

[rule]
actions = []
author = ["513814043"]
description = "Test ESQL Suppress Rule"
enabled = false
exceptions_list = []
false_positives = []
from = "now-360s"
interval = "5m"
language = "esql"
license = ""
max_signals = 100
name = "Test ESQL Suppress Rule"
references = []
related_integrations = []
required_fields = []
revision = 0
risk_score = 21
risk_score_mapping = []
rule_id = "058a8221-5b41-49ad-9e68-5a60fdf977e8"
setup = ""
severity = "low"
severity_mapping = []
tags = []
threat = []
to = "now"
type = "esql"
version = 1

query = '''
from logs* metadata _id, _version, _index |
    WHERE process.name == "test"
'''



[rule.alert_suppression]
group_by = ["process.name"]
missing_fields_strategy = "suppress"

Checklist

• Added a label for the type of pr: bug, enhancement, Rule: New, Rule: Deprecation, Rule: Promote, Rule: Tuning, Hunt: New, or Hunt: Tuning so guidelines can be generated
• Added the meta:rapid-merge label if planning to merge within 24 hours
• Secret and sensitive material has been managed correctly
• Automated testing was updated or added to match the most common scenarios
• Documentation and comments were added for features that require explanation

[protectionsmachine]

Bug - Guidelines

These guidelines serve as a reminder set of considerations when addressing a bug in the code.

Documentation and Context

• Provide detailed documentation (description, screenshots, reproducing the bug, etc.) of the bug if not already documented in an issue.
• Include additional context or details about the problem.
• Ensure the fix includes necessary updates to the release documentation and versioning.

Code Standards and Practices

• Code follows established design patterns within the repo and avoids duplication.
• Code changes do not introduce new warnings or errors.
• Variables and functions are well-named and descriptive.
• Any unnecessary / commented-out code is removed.
• Ensure that the code is modular and reusable where applicable.
• Check for proper exception handling and messaging.

Testing

• New unit tests have been added to cover the bug fix or edge cases.
• Existing unit tests have been updated to reflect the changes.
• Provide evidence of testing and detecting the bug fix (e.g., test logs, screenshots).
• Validate that any rules affected by the bug are correctly updated.
• Ensure that performance is not negatively impacted by the changes.
• Verify that any release artifacts are properly generated and tested.

Additional Checks

• Ensure that the bug fix does not break existing functionality.
• Review the bug fix with a peer or team member for additional insights.
• Verify that the bug fix works across all relevant environments (e.g., different OS versions).
• Confirm that all dependencies are up-to-date and compatible with the changes.

[traut]

huh, nice

[brokensound77]

This should not be handled in the formatter and should be rolled back. This will now blindly strip all empty lists when dumping. Empty lists are valid toml structures and so should not be stripped at all. The proper way to handle this would have been to strip the object itself (the rule) of any unwanted empty lists, similar to:

detection-rules/detection_rules/utils.py

Lines 40 to 45 in df31c00

	class NonelessDict(dict):
	"""Wrapper around dict that doesn't populate None values."""
	def __setitem__(self, key, value):
	if value is not None:
	dict.__setitem__(self, key, value)

or

detection-rules/detection_rules/mixins.py

Lines 31 to 39 in df31c00

	def _strip_none_from_dict(obj: T) -> T:
	"""Strip none values from a dict recursively."""
	if isinstance(obj, dict):
	return {key: _strip_none_from_dict(value) for key, value in obj.items() if value is not None}
	if isinstance(obj, list):
	return [_strip_none_from_dict(o) for o in obj]
	if isinstance(obj, tuple):
	return tuple(_strip_none_from_dict(list(obj)))
	return obj