[Discuss] Supporting ES|QL LOOKUP JOIN on integration data

[andrewkroh]

ES|QL is adding a new LOOKUP JOIN feature (elastic/elasticsearch#116208) that will enable joining data at query time. This feature will be useful in several integrations. For example, the crowdstrike.fdr integration receives two distinct types of data: endpoint events containing a host ID and host metadata events mapping the host ID to metadata about the host. To make these two types of data useful, they need to be joined.

To use the LOOKUP JOIN feature, packages must meet the following requirements:

1. Write data to a regular index that has index.mode: lookup. Enrichment data, such as documents mapping host IDs to host metadata, would be directed to this index by the agent.
2. Be able to evolve the mappings of this enrichment data index (e.g., handle scenarios where you need to change the mapping of a field).

Note: This feature is still in development, and I haven't used it yet, so some of my assumptions here might be incorrect.