Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix application crash when trying to reconnect during application shutdown #1254

Merged
merged 4 commits into from
Oct 1, 2022

Conversation

JcolemanNR
Copy link
Contributor

@JcolemanNR JcolemanNR commented Sep 29, 2022

Description

During application shutdown, if the SendDataOnExit configuration option is enabled, and the minimum timespan for it to trigger has elapsed, the agent attempts to trigger one final harvest cycle before shutting down as well. This works well, but there is an edge case where the collector may return an HTTP response code indicating that the agent should reconnect during these final Harvest cycles.

The current behavior of the agent is to attempt to reconnect even though the application is already shutting down. This behavior can result in unexpected exceptions which can cause an ungraceful application shutdown.

This PR adds a new global boolean to allow application code to behave differently in the event that the application is shutting down. The only application logic that is currently modified based on this boolean is that no agent reconnects will occur during application shutdown. The reasoning for this is that an application is already shutting down, and we are trying to send any last bits of data by manually invoking Harvest cycles for all of the Aggregators, but the semantics of a server invoked agent reconnect are that all event data is thrown out during this period, so there is no point in reconnecting just to wipe out all data, and send no data.

There is no unit or integration test coverage on this code as I was unable to find any existing test coverage for the areas I modified. I was however able to reliably manually reproduce the bug, and observe that it no longer occurs with these proposed changes.

Author Checklist

Reviewer Checklist

  • Perform code review
  • Pull request was adequately tested (new/existing tests, performance tests)
  • Review Changelog

@JcolemanNR JcolemanNR merged commit 06ab230 into main Oct 1, 2022
@JcolemanNR JcolemanNR deleted the jc/fix-crash-on-exit branch October 1, 2022 00:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants