Auto generate ECS fields #141152

nkhristinin · 2022-09-20T18:34:07Z

Auto generate ECS fields from ECS 8.5 branch
Increase field limits for indexes as we did previously here

How I did generate:

Download ecs repo from 8.5 branch
rename folder to ecs
cd ${KIBANA_FOLDER}
cd ..
move/past ecs foder here
cd kibana
node x-pack/plugins/rule_registry/scripts/generate_ecs_fieldmap/index.js

ajosh0504 · 2022-09-20T18:56:00Z

LGTM!

nkhristinin · 2022-09-21T08:38:11Z

@elasticmachine merge upstream

pmuellr · 2022-09-21T17:18:06Z

I'm not seeing any version labels - what stack release are we targetting for this?

nkhristinin · 2022-09-21T17:37:13Z

I'm not seeing any version labels - what stack release are we targetting for this?

@pmuellr The goal is to merge it for 8.5, but the branch not yet created, so I didn't add any specific label

nkhristinin · 2022-09-21T17:38:13Z

@elasticmachine merge upstream

kobelb · 2022-09-21T17:51:12Z

Do we actually need and benefit from all of these ECS fields added to the existing alerts-as-data indices? Prior to us minimizing the number of alert indices that are created as outlined by the future of alerts as data, we should be careful about adding new fields to the alerts-as-data indices. Each field we add to the mappings will increase Elasticsearch's data node heap usage.

This isn't as bad for customers who don't have a large number of Kibana spaces, adding more fields isn't "that bad". For example, if the customer only has a single space, increasing from 1,500 to 2,000 fields will only increase the data node heap usage for alerts indices that are storing 12 months of data from 18.43 MB to 24.58 MB. However, if they have a 1,000 spaces this will increase the data node heap usage from 18.43 GB to 24.58 GB.

rylnd · 2022-09-21T19:24:00Z

Do we actually need and benefit from all of these ECS fields added to the existing alerts-as-data indices?

@kobelb that's a complicated question. This PR does incorporate fields that we do not currently use, as this PR includes:

beta Host Risk fields
general ECS fields added since previous 8.x PR

I think an argument can be made that the beta fields are more numerous than they should be (due to limitations with the ECS tooling), which I'm currently pursuing. That will limit the fields in the first data point to ones that we DO use.

That still leaves the second question, however: if ECS introduces new fields, and users start adopting those fields in their data, when (if at all) do we start supporting that on the solution side?

MikePaquette · 2022-09-22T12:43:07Z

For clarity, can someone confirm how many new fields would actually be added to the alert index mappings with this PR?

Here is some additional background:

ECS 8.5 has 60 more fields than ECS 8.4. (1574 vs 1514)
Of those 60 new fields, 54 of them are related to the new risk fields (6 new fields x 9 nesting places). As @rylnd mentions, we can likely eliminate some/most of the nesting places with updated ECS tooling to reduce this number further.
There are 5 other new ECS fields.
The motivation for this mappings update was driven by the need to add support for the new risk fields to support entity analytics enriching alert documents with entity risk info. We expected that this update would result in adding only the "new in 8.5" ECS fields to the alert mappings, which would have added no more than 60 fields.

The original design principle for the security signals index (now the Alerts index) was that it would always support the full set of ECS fields that were current at the time of release. This was driven by the draft RFC Guidelines for ensuring compatibility between Elastic ECS data producers and consumers #1 which states that the security solution "MUST work properly on any data that is compliant with the latest published version of ECS at the time of the Elastic Stack feature freeze for the app/content."

Over approximately the past year The number of ECS fields had grown as follows:

1.11 - 1171
1.12 - 1239
8.0 - 1269
8.1 - 1315
8.2 - 1503 <- Around here, out of concern for rapidly growing mapping size, we deprioritized the guideline above.
8.3 - 1511
8.4 - 1514
8.5 - 1574

Since approximately the 8.2 timeframe, we've been in an ambiguous state, agreeing to be intentional about adding new fields to keep the alert index mappings from exploding in size, but at the same time, living with the tension of possible user dissatisfaction because we've made an exception to the ECS "contract" to fully support users' ECS-compliant data in solutions (i.e. if user data is in ECS format, it will be added to detection Alerts.)

In essence, we've created a de facto "exclude list" of ECS fields that are not supported in the Alert field mappings. AFAIK, this list is not documented in user-facing documents, and there seems to be some lack of clarity as to how to update alert index mappings with some new ECS fields, but not all ECS fields.

At this point, there seems to be two options:

(default) Continue with the agreement to be intentional about adding fields to the Alert mappings PRO: limits the growth of mapping size of Alerts index. CON: weakens ECS contract with users, causes dev confusion that could result in mapping conflicts or breaking changes for users. Action: Modify the automation in this PR to exclude previously excluded fields. Define process for updating, maintaining, documenting this exclusion list.
Overturn our 8.2-timeframe decision, and again adhere to the principle of including all ECS fields in the Alert index mappings. PRO: Users enjoy the confidence that if their data uses ECS, their fields will get included in Alerts. Much simpler process for solution dev teams - every release, include all the latest ECS mappings. CON: Alerts index mappings grow at the same rate as ECS. Action: Continue with the automation in this PR to add all ECS fields.

nkhristinin · 2022-09-22T15:50:35Z

@MikePaquette
This script generate a new 273 fields:

Show fields

{
  "client": {
    "user": {
      "risk": {
        "calculated_level": {},
        "calculated_score": {},
        "calculated_score_norm": {},
        "static_level": {},
        "static_score": {},
        "static_score_norm": {}
      }
    }
  },
  "container": {
    "cpu": {
      "usage": {}
    },
    "disk": {
      "read": {
        "bytes": {}
      },
      "write": {
        "bytes": {}
      }
    },
    "image": {
      "hash": {
        "all": {}
      }
    },
    "memory": {
      "usage": {}
    },
    "network": {
      "egress": {
        "bytes": {}
      },
      "ingress": {
        "bytes": {}
      }
    }
  },
  "destination": {
    "user": {
      "risk": {
        "calculated_level": {},
        "calculated_score": {},
        "calculated_score_norm": {},
        "static_level": {},
        "static_score": {},
        "static_score_norm": {}
      }
    }
  },
  "dll": {
    "hash": {
      "sha384": {},
      "tlsh": {}
    },
    "pe": {
      "pehash": {}
    }
  },
  "email": {
    "attachments": {
      "file": {
        "extension": {},
        "hash": {
          "md5": {},
          "sha1": {},
          "sha256": {},
          "sha384": {},
          "sha512": {},
          "ssdeep": {},
          "tlsh": {}
        },
        "mime_type": {},
        "name": {},
        "size": {}
      }
    },
    "bcc": {
      "address": {}
    },
    "cc": {
      "address": {}
    },
    "content_type": {},
    "delivery_timestamp": {},
    "direction": {},
    "from": {
      "address": {}
    },
    "local_id": {},
    "message_id": {},
    "origination_timestamp": {},
    "reply_to": {
      "address": {}
    },
    "sender": {
      "address": {}
    },
    "subject": {},
    "to": {
      "address": {}
    },
    "x_mailer": {}
  },
  "faas": {
    "id": {},
    "name": {},
    "version": {}
  },
  "file": {
    "hash": {
      "sha384": {},
      "tlsh": {}
    },
    "pe": {
      "pehash": {}
    }
  },
  "host": {
    "boot": {
      "id": {}
    },
    "pid_ns_ino": {},
    "risk": {
      "calculated_level": {},
      "calculated_score": {},
      "calculated_score_norm": {},
      "static_level": {},
      "static_score": {},
      "static_score_norm": {}
    }
  },
  "log": {
    "syslog": {
      "appname": {},
      "hostname": {},
      "msgid": {},
      "procid": {},
      "structured_data": {},
      "version": {}
    }
  },
  "orchestrator": {
    "cluster": {
      "id": {}
    },
    "resource": {
      "id": {},
      "ip": {},
      "parent": {
        "type": {}
      }
    }
  },
  "process": {
    "entry_leader": {
      "args": {},
      "args_count": {},
      "attested_groups": {
        "name": {}
      },
      "attested_user": {
        "id": {},
        "name": {}
      },
      "command_line": {},
      "entry_meta": {
        "source": {
          "ip": {}
        },
        "type": {}
      },
      "executable": {},
      "group": {
        "id": {},
        "name": {}
      },
      "interactive": {},
      "name": {},
      "parent": {
        "entity_id": {},
        "pid": {},
        "session_leader": {
          "entity_id": {},
          "pid": {},
          "start": {}
        },
        "start": {}
      },
      "pid": {},
      "real_group": {
        "id": {},
        "name": {}
      },
      "real_user": {
        "id": {},
        "name": {}
      },
      "same_as_process": {},
      "saved_group": {
        "id": {},
        "name": {}
      },
      "saved_user": {
        "id": {},
        "name": {}
      },
      "start": {},
      "supplemental_groups": {
        "id": {},
        "name": {}
      },
      "tty": {
        "char_device": {
          "major": {},
          "minor": {}
        }
      },
      "user": {
        "id": {},
        "name": {}
      },
      "working_directory": {}
    },
    "env_vars": {},
    "group_leader": {
      "args": {},
      "args_count": {},
      "command_line": {},
      "entity_id": {},
      "executable": {},
      "group": {
        "id": {},
        "name": {}
      },
      "interactive": {},
      "name": {},
      "pid": {},
      "real_group": {
        "id": {},
        "name": {}
      },
      "real_user": {
        "id": {},
        "name": {}
      },
      "same_as_process": {},
      "saved_group": {
        "id": {},
        "name": {}
      },
      "saved_user": {
        "id": {},
        "name": {}
      },
      "start": {},
      "supplemental_groups": {
        "id": {},
        "name": {}
      },
      "tty": {
        "char_device": {
          "major": {},
          "minor": {}
        }
      },
      "user": {
        "id": {},
        "name": {}
      },
      "working_directory": {}
    },
    "hash": {
      "sha384": {},
      "tlsh": {}
    },
    "interactive": {},
    "parent": {
      "group": {
        "id": {},
        "name": {}
      },
      "group_leader": {
        "entity_id": {},
        "pid": {},
        "start": {}
      },
      "hash": {
        "sha384": {},
        "tlsh": {}
      },
      "interactive": {},
      "pe": {
        "pehash": {}
      },
      "real_group": {
        "id": {},
        "name": {}
      },
      "real_user": {
        "id": {},
        "name": {}
      },
      "saved_group": {
        "id": {},
        "name": {}
      },
      "saved_user": {
        "id": {},
        "name": {}
      },
      "supplemental_groups": {
        "id": {},
        "name": {}
      },
      "tty": {
        "char_device": {
          "major": {},
          "minor": {}
        }
      },
      "user": {
        "id": {},
        "name": {}
      }
    },
    "pe": {
      "pehash": {}
    },
    "previous": {
      "args": {},
      "args_count": {},
      "executable": {}
    },
    "real_group": {
      "id": {},
      "name": {}
    },
    "real_user": {
      "id": {},
      "name": {}
    },
    "saved_group": {
      "id": {},
      "name": {}
    },
    "saved_user": {
      "id": {},
      "name": {}
    },
    "session_leader": {
      "args": {},
      "args_count": {},
      "command_line": {},
      "executable": {},
      "group": {
        "id": {},
        "name": {}
      },
      "interactive": {},
      "name": {},
      "parent": {
        "entity_id": {},
        "pid": {},
        "session_leader": {
          "entity_id": {},
          "pid": {},
          "start": {}
        },
        "start": {}
      },
      "pid": {},
      "real_group": {
        "id": {},
        "name": {}
      },
      "real_user": {
        "id": {},
        "name": {}
      },
      "same_as_process": {},
      "saved_group": {
        "id": {},
        "name": {}
      },
      "saved_user": {
        "id": {},
        "name": {}
      },
      "start": {},
      "supplemental_groups": {
        "id": {},
        "name": {}
      },
      "tty": {
        "char_device": {
          "major": {},
          "minor": {}
        }
      },
      "user": {
        "id": {},
        "name": {}
      },
      "working_directory": {}
    },
    "supplemental_groups": {
      "id": {},
      "name": {}
    },
    "tty": {
      "char_device": {
        "major": {},
        "minor": {}
      },
      "columns": {},
      "rows": {}
    },
    "user": {
      "id": {},
      "name": {}
    }
  },
  "server": {
    "user": {
      "risk": {
        "calculated_level": {},
        "calculated_score": {},
        "calculated_score_norm": {},
        "static_level": {},
        "static_score": {},
        "static_score_norm": {}
      }
    }
  },
  "service": {
    "node": {
      "role": {},
      "roles": {}
    },
    "origin": {
      "node": {
        "role": {},
        "roles": {}
      }
    },
    "target": {
      "node": {
        "role": {},
        "roles": {}
      }
    }
  },
  "source": {
    "user": {
      "risk": {
        "calculated_level": {},
        "calculated_score": {},
        "calculated_score_norm": {},
        "static_level": {},
        "static_score": {},
        "static_score_norm": {}
      }
    }
  },
  "threat": {
    "enrichments": {
      "indicator": {
        "file": {
          "hash": {
            "sha384": {},
            "tlsh": {}
          },
          "pe": {
            "pehash": {}
          }
        }
      },
      "matched": {
        "occurred": {}
      }
    },
    "feed": {
      "dashboard_id": {},
      "description": {},
      "name": {},
      "reference": {}
    },
    "indicator": {
      "file": {
        "hash": {
          "sha384": {},
          "tlsh": {}
        },
        "pe": {
          "pehash": {}
        }
      }
    }
  },
  "user": {
    "changes": {
      "risk": {
        "calculated_level": {},
        "calculated_score": {},
        "calculated_score_norm": {},
        "static_level": {},
        "static_score": {},
        "static_score_norm": {}
      }
    },
    "effective": {
      "risk": {
        "calculated_level": {},
        "calculated_score": {},
        "calculated_score_norm": {},
        "static_level": {},
        "static_score": {},
        "static_score_norm": {}
      }
    },
    "risk": {
      "calculated_level": {},
      "calculated_score": {},
      "calculated_score_norm": {},
      "static_level": {},
      "static_score": {},
      "static_score_norm": {}
    },
    "target": {
      "risk": {
        "calculated_level": {},
        "calculated_score": {},
        "calculated_score_norm": {},
        "static_level": {},
        "static_score": {},
        "static_score_norm": {}
      }
    }
  }
}

MikePaquette · 2022-09-22T16:44:44Z

Decision:

Short term: Add only the 12 new risk fields (6 fields x 2 nesting places [host.*, user.*]) to the Alert index mapptings for 8.5
Next: Option 1 - Modify the automation in this PR to exclude previously excluded fields. Define process for updating, maintaining, documenting this exclusion list, and for adding selected new ECS fields.

simianhacker · 2022-09-22T19:08:41Z

If end up doing a subset of fields, at a minimum we need:

host.*
cloud.*
orchestrator.*
labels.*
tags

rylnd · 2022-09-22T22:08:51Z

If end up doing a subset of fields, at a minimum we need:

@simianhacker is this specific to the risk.* nestings, or are you talking generally about observability's field needs (relative to what's missing from current mappings)?

More explicitly: are you saying that there are NEW ECS fields that o11y needs in 8.5, or are you just requesting that we not remove any of the existing fields?

simianhacker · 2022-09-22T22:15:54Z

@rylnd I'm saying at a minimum, O11y needs those "top level" fields (objects) added to the mappings for Alert-As-Data.

nkhristinin · 2022-09-23T08:27:37Z

I updated PR, with only risk fields and some updates for fields that @simianhacker mention

nkhristinin · 2022-09-23T15:43:49Z

@elasticmachine merge upstream

kibana-ci · 2022-09-23T17:06:10Z

💚 Build Succeeded

Buildkite Build
Commit: 54b8f2d

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`observability`	525.4KB	526.5KB	+1.1KB
`securitySolution`	6.6MB	6.6MB	+1.1KB
`triggersActionsUi`	1.1MB	1.1MB	+1.1KB
total			+3.4KB

History

💔 Build #75150 failed 7032a3e
💚 Build #74567 succeeded e5660c0
💛 Build #74441 was flaky cd06cb2
💔 Build #74322 failed 3f81ab7
💔 Build #74141 failed 76e2d93

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

rylnd

LGTM, thanks for following up here!

* Auto generate ecs fields * Update field limits * Update ecs fields Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> (cherry picked from commit e039e3e)

kibanamachine · 2022-09-24T12:29:53Z

💚 All backports created successfully

Status	Branch	Result
✅	8.5

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

* Auto generate ecs fields * Update field limits * Update ecs fields Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> (cherry picked from commit e039e3e) Co-authored-by: Khristinin Nikita <nikita.khristinin@elastic.co>

Auto generate ecs fields

76e2d93

nkhristinin requested review from a team as code owners September 20, 2022 18:34

nkhristinin added the release_note:skip Skip the PR/issue when compiling release notes label Sep 20, 2022

nkhristinin requested review from rylnd and ajosh0504 September 20, 2022 18:37

ajosh0504 approved these changes Sep 20, 2022

View reviewed changes

kibanamachine and others added 2 commits September 21, 2022 02:38

Merge branch 'main' into ecs-fields

3f81ab7

Update field limits

cd06cb2

simianhacker self-requested a review September 21, 2022 16:03

nkhristinin added the v8.5.0 label Sep 21, 2022

Merge branch 'main' into ecs-fields

e5660c0

nkhristinin added the backport:prev-minor Backport to (9.0) the previous minor version (i.e. one version back from main) label Sep 22, 2022

Update ecs fields

7032a3e

Merge branch 'main' into ecs-fields

54b8f2d

rylnd approved these changes Sep 23, 2022

View reviewed changes

nkhristinin merged commit e039e3e into elastic:main Sep 24, 2022

kibanamachine added the v8.6.0 label Sep 24, 2022

kibanamachine mentioned this pull request Sep 24, 2022

[8.5] Auto generate ECS fields (#141152) #141726

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Auto generate ECS fields #141152

Auto generate ECS fields #141152

nkhristinin commented Sep 20, 2022 •

edited by kibanamachine

Loading

ajosh0504 commented Sep 20, 2022

nkhristinin commented Sep 21, 2022

pmuellr commented Sep 21, 2022

nkhristinin commented Sep 21, 2022

nkhristinin commented Sep 21, 2022

kobelb commented Sep 21, 2022

rylnd commented Sep 21, 2022

MikePaquette commented Sep 22, 2022

nkhristinin commented Sep 22, 2022 •

edited

Loading

MikePaquette commented Sep 22, 2022

simianhacker commented Sep 22, 2022

rylnd commented Sep 22, 2022

simianhacker commented Sep 22, 2022

nkhristinin commented Sep 23, 2022

nkhristinin commented Sep 23, 2022

kibana-ci commented Sep 23, 2022

rylnd left a comment

kibanamachine commented Sep 24, 2022

Auto generate ECS fields #141152

Auto generate ECS fields #141152

Conversation

nkhristinin commented Sep 20, 2022 • edited by kibanamachine Loading

ajosh0504 commented Sep 20, 2022

nkhristinin commented Sep 21, 2022

pmuellr commented Sep 21, 2022

nkhristinin commented Sep 21, 2022

nkhristinin commented Sep 21, 2022

kobelb commented Sep 21, 2022

rylnd commented Sep 21, 2022

MikePaquette commented Sep 22, 2022

nkhristinin commented Sep 22, 2022 • edited Loading

MikePaquette commented Sep 22, 2022

simianhacker commented Sep 22, 2022

rylnd commented Sep 22, 2022

simianhacker commented Sep 22, 2022

nkhristinin commented Sep 23, 2022

nkhristinin commented Sep 23, 2022

kibana-ci commented Sep 23, 2022

💚 Build Succeeded

Metrics [docs]

Async chunks

History

rylnd left a comment

Choose a reason for hiding this comment

kibanamachine commented Sep 24, 2022

💚 All backports created successfully

Questions ?

nkhristinin commented Sep 20, 2022 •

edited by kibanamachine

Loading

nkhristinin commented Sep 22, 2022 •

edited

Loading