Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FIXED] Ghost consumers after failed meta proposal #6088

Merged
merged 1 commit into from
Nov 7, 2024

Conversation

MauriceVanVeen
Copy link
Member

If the cc.meta.Propose fails, for example due to restarts, the sa.consumers[ca.Name] = ca would have staged a consumer that will never be successfully added. That consumer would then become part of the next snapshot (for example during shutdown), which meant that that server had a ghost consumer it was unable to clean up.

This does not fully solve the ghost consumers issue, but this should contribute to the final solution.

Signed-off-by: Maurice van Veen github@mauricevanveen.com

@MauriceVanVeen MauriceVanVeen requested a review from a team as a code owner November 7, 2024 16:54
Copy link
Member

@derekcollison derekcollison left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should there be a test here that shows the change in behavior?

@derekcollison
Copy link
Member

This test seems to fail now on this PR - TestJetStreamClusterConsumerLeak

@MauriceVanVeen
Copy link
Member Author

Should there be a test here that shows the change in behavior?

There was already TestJetStreamClusterMaxConsumersMultipleConcurrentRequests that partially covers this.
Have extended it a little to also check sa.pendingConsumers gets properly cleared.

Signed-off-by: Maurice van Veen <github@mauricevanveen.com>
@MauriceVanVeen MauriceVanVeen force-pushed the maurice/fix-ghost-consumers branch from 51e0ff5 to 3c0afd2 Compare November 7, 2024 19:54
@MauriceVanVeen
Copy link
Member Author

This test seems to fail now on this PR - TestJetStreamClusterConsumerLeak

At least in CI it seems, locally it passes consistently for me.

@derekcollison derekcollison self-requested a review November 7, 2024 22:55
Copy link
Member

@derekcollison derekcollison left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@derekcollison derekcollison merged commit 73a481d into main Nov 7, 2024
5 checks passed
@derekcollison derekcollison deleted the maurice/fix-ghost-consumers branch November 7, 2024 22:58
neilalexander added a commit that referenced this pull request Nov 12, 2024
Includes the following:

* Some tweaks to the NRG test helpers
* #6055
* #6061
* #6065 
* #6041 (but with `math/rand`
instead of `math/rand/v2` due to an older Go version in CI for 2.10.x)
* #6066
* #6067
* #6069
* #6075
* #6082
* #6087
* #6086
* #6088
* #6089
* #6092
* #6096
* #6098
* #6097
* #6105
* #6104
* #6106
* #6109
* #6111
* #6112

Signed-off-by: Neil Twigg <neil@nats.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants