Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ingest: introduce apm_data_stream_migration #5768

Merged
merged 1 commit into from
Jul 21, 2021
Merged

ingest: introduce apm_data_stream_migration #5768

merged 1 commit into from
Jul 21, 2021

Conversation

axw
Copy link
Member

@axw axw commented Jul 20, 2021 •
edited
Loading

Motivation/summary

Introduce a new ingest pipeline, apm_data_stream_migration, which can be used in a reindex operation to set data_stream fields on events and route them to appropriate data streams.

Checklist

How to test these changes

  1. Run apm-server standalone, send some data of various types
  2. Install the "apm" integration (no need to add it to an agent/policy etc., just need the index templates to be created)
  3. Reindex apm-* using this pipeline, excluding processor.event: onboarding
  4. Check the events have data_stream.* fields added, and they are routed to the correct data streams

Related issues

Closes #4854

@axw axw force-pushed the pipeline-migrate-datastream branch 2 times, most recently from 62bb69d to 56d785c Compare July 20, 2021 09:33
@apmmachine
Copy link
Contributor

apmmachine commented Jul 20, 2021 •
edited
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2021-07-20T12:59:45.436+0000

  • Duration: 50 min 0 sec

  • Commit: 38ec1d5

Test stats 🧪

Test Results
Failed 0
Passed 5941
Skipped 14
Total 5955

Trends 🧪

Image of Build Times

Image of Tests

@axw axw force-pushed the pipeline-migrate-datastream branch from 56d785c to 38ec1d5 Compare July 20, 2021 12:59
@axw axw marked this pull request as ready for review July 20, 2021 12:59
@axw axw requested a review from a team July 20, 2021 12:59
stuartnelson3
stuartnelson3 approved these changes Jul 20, 2021
View reviewed changes
Copy link
Contributor

@stuartnelson3 stuartnelson3 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not experienced with pipelines but the script code makes sense 👍

@axw axw merged commit 4f4dca2 into elastic:master Jul 21, 2021
@axw axw deleted the pipeline-migrate-datastream branch July 21, 2021 01:08
mergify bot pushed a commit that referenced this pull request Jul 21, 2021
@axw
ingest: introduce apm_data_stream_migration (#5768)
20ff337 
(cherry picked from commit 4f4dca2)

# Conflicts:
#	changelogs/head.asciidoc
@mergify mergify bot mentioned this pull request Jul 21, 2021
Merged
@axw axw mentioned this pull request Jul 21, 2021
Open
axw added a commit that referenced this pull request Jul 21, 2021
@mergify @axw
[7.x] ingest: introduce apm_data_stream_migration (backport #5768) (#…
526d4ec 
…5772)

* ingest: introduce apm_data_stream_migration (#5768)

(cherry picked from commit 4f4dca2)

# Conflicts:
#	changelogs/head.asciidoc

* Delete head.asciidoc

Co-authored-by: Andrew Wilkins <axw@elastic.co>
@axw axw mentioned this pull request Aug 30, 2021
Closed
@stuartnelson3 stuartnelson3 self-assigned this Aug 31, 2021
@stuartnelson3
Copy link
Contributor

Confirmed with BC3

@stuartnelson3 stuartnelson3 removed their assignment Aug 31, 2021
@viewable viewable mentioned this pull request Aug 16, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@stuartnelson3 stuartnelson3 stuartnelson3 approved these changes

Assignees
No one assigned
Labels
test-plan test-plan-ok v7.15.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

apmpackage: install a pipeline for migrating pre-datastream indices
4 participants
@axw @apmmachine @stuartnelson3 @simitt