Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SQS messages deleted without being processed #360

Closed
azenk opened this issue Feb 10, 2021 · 1 comment · Fixed by #364
Closed

SQS messages deleted without being processed #360

azenk opened this issue Feb 10, 2021 · 1 comment · Fixed by #364
Assignees
Labels
Type: Bug Something isn't working Type: Question All types of questions to/from customers

Comments

@azenk
Copy link
Contributor

azenk commented Feb 10, 2021

When several nodes are being terminated at once, we've observed that some of the messages are deleted from the queue without being processed.

This seems to be due to NTH retrieving up to two messages from SQS, only processing the first but passing both into asgTerminationToInterruptionEvent where both messages are deleted.

@jillmon jillmon added the Type: Question All types of questions to/from customers label Feb 11, 2021
@haugenj haugenj added the Type: Bug Something isn't working label Feb 12, 2021
@haugenj
Copy link
Contributor

haugenj commented Feb 12, 2021

This is a good callout. The easy fix would be to change the request to accept a max of 1 message, but I think we should go the extra mile and parse them into multiple events similar to how we're doing in scheduled-event-monitor.go.

@azenk are you interested in picking this up? Otherwise I can take it on sometime next week

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Type: Bug Something isn't working Type: Question All types of questions to/from customers
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants