EP Meta Telemetry Perf

[pjhampton]

Summary

Refactoring some parts of the EP Meta telemetry + making performance improvements.

Protections + Endpoint are now happy with the payload shape. We will send this back daily to make improvements to endpoint/protections and identify UX issues or bugs.

Checklist

Delete any items that are not applicable to this PR.

• Unit or functional tests were updated or added to match the most common scenarios

For maintainers

• This was checked for breaking API changes and was labeled appropriately

[pjhampton]

@elasticmachine merge upstream

[pjhampton]

@elasticmachine merge upstream

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[pjhampton]

@elasticmachine merge upstream

[donaherc]

LGTM, but will definitely defer to other security app devs for additional feedback

[pjhampton]

@elasticmachine merge upstream

[jonathan-buttner]

Great comments! Left some ideas for avoiding the casting.

[jonathan-buttner]

Just responding to your question from slack here, I know you didn't change this but yeah if you have time to make a few changes here that'd be great.

I believe it's possible for endpointMetricsResponse to be undefined based on the return value of the this.fetchEndpointData() call. Below I think we can change the if to if (endpointMetricsResponse?.aggregation to address that

I think if we could remove the cast to unknown that'd be awesome. I think we can achieve that by adding something like this:

interface MetricsType {
  key: string;
  doc_count: number;
  latest_metrics: EndpointMetricHits;
}

const endpointAgentsAggs = endpointMetricsResponse.aggregations
      .endpoint_agents as AggregationsMultiBucketAggregate<MetricsType>;

    const endpointMetrics = endpointAgentsAggs.buckets.map((epMetrics) => {
      return {
        endpoint_agent: epMetrics.latest_metrics.hits.hits[0]._source.agent.id,
        endpoint_version: epMetrics.latest_metrics.hits.hits[0]._source.agent.version,
        endpoint_metrics: epMetrics.latest_metrics.hits.hits[0]._source,
      };

As a side note it looks like EndpointMetricsAggregation might need total I think that is already typed as part of the SearchResponse type, so we might be able to get rid of that part.

export interface EndpointMetricsAggregation {
  hits: {
    total: { value: number };
  };

[pjhampton]

@elasticmachine merge upstream

[pjhampton]

@elasticmachine merge upstream

[michaelolo24]

Is the map just to cast the values, or should this be a filter of any null values? If the former could we just remove the map like below? Also, can the pPolicy.inputs array be empty or the config null instead of undefined?

packagePolicies
  .forEach((pPolicy as PolicyData) => {
    if (pPolicy.inputs[0].config !== undefined) { // <-- Also, can this be null too?
      pPolicy.inputs.forEach((input) => {
        if (
          input.type === FLEET_ENDPOINT_PACKAGE &&
          input.config !== undefined &&
          policyInfo !== undefined
        ) {
          endpointPolicyCache.set(policyInfo, pPolicy);
        }
      });
    }
  });

[pjhampton]

Is the map just to cast the values, or should this be a filter of any null values?

It is the former - I am just casting. There shouldn't be any null values (but anything is possible).

Also, can the pPolicy.inputs array be empty or the config null instead of undefined?

I haven't seen it - as these are package policies instead of agent policies I think they should always have an input. But I'm running with the assumption that it is possible.

[michaelolo24]

Would a reduce combining these two be easier? So:

failedPolicy?._source.Endpoint.policy.applied.actions.reduce((memo, action) => {
	if (action.status !== 'success') memo.push(action);
	return memo;
}, []);

[pjhampton]

I think this is a great suggestion, but I prefer how my implementation reads by functional chaining. Hope that is ok :-)

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: d0cf37d
• Storybooks Preview

Metrics [docs]

✅ unchanged

History

• 💚 Build #139381 succeeded e451818
• 💚 Build #139275 succeeded dba608e
• 💚 Build #139029 succeeded 7877da2
• 💚 Build #139019 succeeded d783013
• 💚 Build #139010 succeeded 65343d8

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @pjhampton

[kibanamachine]

💚 Backport successful

Status	Branch	Result
✅	7.14
✅	7.x

The backport PRs will be merged automatically after passing CI.