[Meta] Migrate telemetry usage_collectors to the new ES Client

[TinaHeiligers]

Support for the legacy Elasticsearch client will be removed in 7.15.

The Telemetry team (now the Core team) added support for using the new ES client in 7.11 to allow plugins that use a usage_collector to begin migrating over.

In 7.13 7.12, the legacy client will be completely removed from collectors' fetch context. In order to continue report usage data, usage collectors that make calls to elasticsearch for saving/retrieving telemetric data need to use the new elasticsearch client.

The deadline for collector migrations is 7.13 7.12 and blocks completion of the migration of the usage_collection plugin.

Collectors that need to be migrated:

collector	migration status
kql_telemetry (#86585)	✅
search	✅
sample_data	✅
canvas	✅
reporting	✅
rollup	✅
security_solution	✅
spaces_usage_collector	✅
Uptime: kibana_telemetry_adapter	✅
xpack_legacy: settings	✅
kibana_usage_collector	✅
TSVB validation_telemetry_service	✅
‘vis_type_vega`	✅
‘visualizations`	✅

Please We need to focus on migrating the collectors as a priority to meet the 7.12 deadline.

The telemetry team migrated the telemetry-related collectors already, and have examples of how you can follow suite:

1. cluster stats
2. nodes usage
3. license info

Thank you to the vis-team for already having done the work!

We ask that each team link back to this issue with an expected completion timeframe and update the status of the collector(s) in question once addressed.

We ask that each team please review the PR's linked backed to this issue so that we can update the status of the collector(s) once addressed.

Migrations required by CODEOWNERS:

kibana-core

• kibana_usage_collector
• xpack_legacy: settings

kibana-core-ui

• sample_data

@elastic/kibana-core-ui

kibana-app-services

• kql_telemetry
• search

@elastic/kibana-app-services

kibana-presentation

• canvas

@elastic/kibana-presentation

kibana-reporting-services

• reporting

@elastic/kibana-reporting-services

es-ui

• rollup

@elastic/es-ui

siem/endpoint-app-team

• security_solution

@elastic/endpoint-app-team

kibana-security

• spaces_usage_collector

@elastic/kibana-security

uptime

• Uptime: kibana_telemetry_adapter

@elastic/uptime

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[TinaHeiligers]

@afharo I've removed the monitoring usage collector from the dependencies list. After decoupling monitoring from kibana usage collection and supplying the KibanaRequest in the collector's fetch context, the only thing that the monitoring collector is using from the fetch context is the KibanaRequest.

Please correct me if I'm wrong here but after digging through the code again, it looks to me as though the monitoring plugin itself provides callCluster. If that is the case, then we don't need to worry about that breaking when we remove support for the legacy client in the telemetry-related plugins. Am I right?

I've pinged the monitoring team but I don't expect to get a response any time soon 😄

[afharo]

@TinaHeiligers I think your assumptions are correct! AFAIK, the Monitoring plugin maintains its own client because of the additional permissions the monitoring user has. They are scoping the client via the KibanaRequest param. I doubt they'll be affected when we remove the callCluster property in the fetch context 👍

[TinaHeiligers]

I doubt they'll be affected when we remove the callCluster property in the fetch context 👍

@afharo I did have to change the reference to the callCluster type in the monitoring stats collectors from StatsCollectionConfig['callCluster'] to LegacyAPICaller in the final PR for handling #83124. Removing callCluster also involved removing it from the StatsCollectionConfig in the telemetry collection manager.

[afharo]

@TinaHeiligers That is great! Thanks!

Do you think we can close this issue now?

[TinaHeiligers]

All collectors have been migrated.