Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Redistribute spans when the number of peers changes #1258

Closed
kentquirk opened this issue Jul 29, 2024 · 0 comments · Fixed by #1268
Closed

Redistribute spans when the number of peers changes #1258

kentquirk opened this issue Jul 29, 2024 · 0 comments · Fixed by #1268
Assignees
@VinozzZ
Labels
type: enhancement New feature or request
Milestone
v2.8

Comments

@kentquirk
Copy link
Contributor

Is your feature request related to a problem? Please describe.

When the number of peers changes, the decision node for traces in flight may move to a new node. We should also evaluate spans that are currently stored and re-evaluate if they need to move to a new peer. This will enable cluster size changes to not break telemetry in flight (different decisions causing missing spans in Honeycomb).

Describe the solution you'd like

Describe alternatives you've considered

Additional context
@kentquirk kentquirk added the type: enhancement New feature or request label Jul 29, 2024
@kentquirk kentquirk added this to the v2.8 milestone Jul 29, 2024
@VinozzZ VinozzZ mentioned this issue Aug 12, 2024
Merged
@VinozzZ VinozzZ self-assigned this Aug 12, 2024
VinozzZ added a commit that referenced this issue Aug 14, 2024
@VinozzZ
feat: redistribute traces on peer membership changes (#1268)
1aad5b2 
<!--
Thank you for contributing to the project! 💜
Please make sure to:
- Chat with us first if this is a big change
  - Open a new issue (or comment on an existing one)
- We want to make sure you don't spend time implementing something we
might have to say No to
- Add unit tests
- Mention any relevant issues in the PR description (e.g. "Fixes #123")

Please see our [OSS process
document](https://github.com/honeycombio/home/blob/main/honeycomb-oss-lifecycle-and-practices.md#)
to get an idea of how we operate.
-->

## Which problem is this PR solving?

Draining incoming and peer queue on peer membership changes
#1258 

## Short description of the changes

A `RedistributeNotifier` is introduced in the collector in this PR. 
- It's initialized on refinery start up
- When a peer membership change is broadcasted, the notifier immediately
signals the `collect()` goroutine to start the redistribution process
- When multiple peer membership changes happen in a row, it ignores the
signal if there's already a redistribution process running
@VinozzZ VinozzZ closed this as completed Aug 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@VinozzZ VinozzZ

Labels
type: enhancement New feature or request
Projects
None yet
Milestone
v2.8
Development

Successfully merging a pull request may close this issue.

feat: redistribute traces on peer membership changes honeycombio/refinery
2 participants
@kentquirk @VinozzZ