Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Upgrade Assistant] Handle migration of legacy APM indices #91875

Open
alisonelizabeth opened this issue Feb 18, 2021 · 5 comments
Open

[Upgrade Assistant] Handle migration of legacy APM indices #91875

alisonelizabeth opened this issue Feb 18, 2021 · 5 comments
Labels
enhancement New value added to drive a business result Feature:Upgrade Assistant Team:Kibana Management Dev Tools, Index Management, Upgrade Assistant, ILM, Ingest Node Pipelines, and more

Comments

@alisonelizabeth
Copy link
Contributor

alisonelizabeth commented Feb 18, 2021

This work is necessary for the 7.x-8 upgrade. This work has been deferred to 8.x. The below information might be partially or completely obsolete.

Background
APM Server is undergoing some fairly significant changes over the next few 7.x releases, moving from being a standalone service to one that runs under Fleet. As part of that effort we are migrating to use data streams instead of raw indices. We also anticipate some breaking changes to the schema, e.g. combining multiple numerical fields into a single aggregate_metric_double field.

For the remainder of 7.x there will be the option to use either indices or data streams, but in 8.0 we will require Fleet and data streams will be the only option. In 8.0, the APM app in Kibana will query only the new data streams. Because only the new data streams will be queried, we should provide an option for users to migrate their old data.

Requirements
The Upgrade Assistant will need to provide a means of reindexing old data (similar to the 6-7 upgrade):

  1. Upgrade Assistant checks for legacy APM indices, and checks that the "apm" integration has been installed by Fleet (specifically, we need to make sure the index templates are in place).
  2. Upgrade Assistant reindexes with a Painless script provided by the APM team. To keep things simple, I think we could just add this to our Fleet integration package and have it always installed. Given that in step 1 we've established the package is installed, Upgrade Assistant can just reference the installed pipeline.
@alisonelizabeth alisonelizabeth added enhancement New value added to drive a business result Team:Kibana Management Dev Tools, Index Management, Upgrade Assistant, ILM, Ingest Node Pipelines, and more Feature:Upgrade Assistant labels Feb 18, 2021
@elasticmachine
Copy link
Contributor

Pinging @elastic/es-ui (Team:Elasticsearch UI)

@axw
Copy link
Member

axw commented Feb 22, 2021

Upgrade Assistant checks for legacy indices

By default apm-server indexes data into indices matching apm-*, but this can be customised. So I suppose this should be customisable in Upgrade Assistant too.

In addition to checking the index name, we should probably search on observer.type: apm-server

@axw
Copy link
Member

axw commented Jul 21, 2021

@alisonelizabeth FYI, I've just merged a change in apm-server to introduce a new ingest pipeline called apm_data_stream_migration: elastic/apm-server#5768. This can be used to reindex docs in legacy APM indices to set data_stream.* fields and route them to the appropriate data streams.

@alisonelizabeth
Copy link
Contributor Author

Great! Thanks @axw. I will likely follow-up with you with questions once I'm able to start working on this.

@cjcenizal cjcenizal self-assigned this Aug 16, 2021
@cjcenizal
Copy link
Contributor

@axw and I chatted about this over Slack on August 25. Andrew informed me that at the time the team wasn't sure if they'd need support for upgrading/reindexing indices to data streams in 8.0. APM UI will continue to read indices in 8.0 instead of introducing a hard break, but it's currently unclear how far into the future they'll extend that support. We agreed to defer working on this until they have more clarity on this.

@cjcenizal cjcenizal removed their assignment Sep 28, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New value added to drive a business result Feature:Upgrade Assistant Team:Kibana Management Dev Tools, Index Management, Upgrade Assistant, ILM, Ingest Node Pipelines, and more
Projects
None yet
Development

No branches or pull requests

4 participants