Fix application crash when trying to reconnect during application shutdown #1254
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
During application shutdown, if the SendDataOnExit configuration option is enabled, and the minimum timespan for it to trigger has elapsed, the agent attempts to trigger one final harvest cycle before shutting down as well. This works well, but there is an edge case where the collector may return an HTTP response code indicating that the agent should reconnect during these final Harvest cycles.
The current behavior of the agent is to attempt to reconnect even though the application is already shutting down. This behavior can result in unexpected exceptions which can cause an ungraceful application shutdown.
This PR adds a new global boolean to allow application code to behave differently in the event that the application is shutting down. The only application logic that is currently modified based on this boolean is that no agent reconnects will occur during application shutdown. The reasoning for this is that an application is already shutting down, and we are trying to send any last bits of data by manually invoking Harvest cycles for all of the Aggregators, but the semantics of a server invoked agent reconnect are that all event data is thrown out during this period, so there is no point in reconnecting just to wipe out all data, and send no data.
There is no unit or integration test coverage on this code as I was unable to find any existing test coverage for the areas I modified. I was however able to reliably manually reproduce the bug, and observe that it no longer occurs with these proposed changes.
Author Checklist
Reviewer Checklist