[cmd/opampsupervisor]: Supervisor waits for configurable healthchecks to report remote config status

[srikanthccv]

Description:

This pull request addresses the remote config status reporting issue discussed in #21079 by introducing the following options to the Agent config:

1. config_apply_timeout: config update is successful if we receive a healthy status and then observe no failure updates for the entire duration of the timeout period; otherwise, failure is reported.

Link to tracking Issue: #21079

Testing: Added e2e test

Documentation:

[srikanthccv]

Looking into unit-test failures

[evan-bradley]

@srikanthccv With #35488 we're going to have the ability for the Collector to report its own health through the OpAMP extension. Do you think we could use this to get rid of the successful_health_checks option and rely on config_apply_timeout for success/failure reporting? Something like: if a success is reported, wait for the duration set in config_apply_timeout before reporting success to the server in case that status changes in that time. If a failure is reported, immediately report a failure, and if no status is reported by the end of the timeout period, report a failure.

[srikanthccv]

I have a question. Let's consider a scenario involving asynchronous errors resulting from a configuration update. The supervisor has applied the new effective configuration and restarted the agent. The collector initially starts without issues, but shortly after, some component reports an asynchronous error, which causes the collector to shut down. In this case, should we categorize this as a successful or failed configuration update? would extension not report any status in this situation?

[evan-bradley]

In this case, should we categorize this as a successful or failed configuration update?

I think this depends on the timeout duration. If it's within the timeout, it would be a failed configuration update, otherwise we would have already that the config was successfully updated and report the issue separately since we don't directly know whether it was related to the config update or not.

would extension not report any status in this situation?

The healthcheck extension and OpAMP extension use the same health reporting mechanisms. Currently both only report errors that cause the Collector to be shutdown.

[srikanthccv]

Went through the service and collector initialization code for a better understanding. IIUC, In the case of an asynchronous error, the extension would initially report a healthy status, but then quickly switch to an unhealthy status during the shutdown process. Can we fully rely on the initial healthy status report? If we do, we might incorrectly flag an update as successful when it's actually failing. What do you think?

[srikanthccv]

We should be able to get rid of the successful_health_checks. We would consider an update successful only if we receive a healthy status and then observe no failure updates for the entire duration of the timeout period?

[evan-bradley]

We would consider an update successful only if we receive a healthy status and then observe no failure updates for the entire duration of the timeout period?

That was my thought as well. I think we can mitigate the case you outlined if we do this.

[srikanthccv]

Great, I'll proceed with implementing the change.

[djaglowski]

Please resolve conflicts and we'll get this merged