[cmd/opampsupervisor] Supervisor fails healthcheck with bootstrap config

[BinaryFissionGames]

Component(s)

cmd/opampsupervisor

What happened?

Description

When running the bootstrap config, the agent doesn't have the healthcheck configuration configured. After bootstrapping, the collector still uses the bootstrap config, which causes the healthcheck to fail every health check interval.

Steps to Reproduce

Start supervisor. Wait for health check to occur.

Expected Result

Agent is considered healthy.

Actual Result

2024-03-20T16:18:56.705-0400    ERROR   supervisor/supervisor.go:673    Agent is not healthy    {"error": "Get \"http://localhost:57806\": dial tcp [::1]:57806: connect: connection refused"}
github.com/open-telemetry/opentelemetry-collector-contrib/cmd/opampsupervisor/supervisor.(*Supervisor).healthCheck
        /Users/brandonjohnson/git_repos/opentelemetry-collector-contrib/cmd/opampsupervisor/supervisor/supervisor.go:673
github.com/open-telemetry/opentelemetry-collector-contrib/cmd/opampsupervisor/supervisor.(*Supervisor).runAgentProcess
        /Users/brandonjohnson/git_repos/opentelemetry-collector-contrib/cmd/opampsupervisor/supervisor/supervisor.go:733
github.com/open-telemetry/opentelemetry-collector-contrib/cmd/opampsupervisor/supervisor.NewSupervisor.func1
        /Users/brandonjohnson/git_repos/opentelemetry-collector-contrib/cmd/opampsupervisor/supervisor/supervisor.go:169

Collector version

5bf424d

Environment information

No response

OpenTelemetry Collector configuration

No response

Log output

No response

Additional context

No response

[github-actions]

Pinging code owners:

• cmd/opampsupervisor: @evan-bradley @atoulme @tigrannajaryan

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[Frapschen]

It appears that there is a network error. Can you confirm if your agent is exposing the 57806 port?

[BinaryFissionGames]

Well, that's exactly it. The agent does not expose that port, but it should, since the config is generated and run by the supervisor.

receivers:
  otlp:
    protocols:
      http:
        endpoint: "localhost:58262"
exporters:
  debug:
    verbosity: basic

extensions:
  opamp:
    instance_uid: "01HSERAG6T06AFVGQT5ZYC0GEK"
    server:
      ws:
        endpoint: "ws://localhost:58263/v1/opamp"
        tls:
          insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [debug]
  extensions: [opamp]

I think the "no-op" config/bootstrap config that the collector runs just needs the health extension config merged into it.

I feel like part of the problem is that the bootstrap config overwrites the collector config (they use the same file), so this is always the initial config for the collector.

Maybe that should be its own separate issue, though.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• cmd/opampsupervisor: @evan-bradley @atoulme @tigrannajaryan

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[cforce]

It is a bit frustrating that the supervisor generates an effective.yaml either before or after, or in both cases, it connects to Opamp Backend and there is not yet a cfg assigned. This embedded configuration lacks at least a proper health check, so the supervisor does not recognize that the collector is up and running properly. As a result, this causes continuous error logs or even restarts of the collector. In offline mode (when the supervisor is not connected to any OpAMP server), the supervisor should come with a basic working collector configuration.

I tried wo workaround by setting up a configuration in the supervisor's YAML file for the collector with a correct path which merged the missing healhtcheck, but this also does not work
My expectation is that if the configuration path is not set, the supervisor should at least create an effective.yaml that meets its own requirements based on static embedded default cfg.
And as requested as (optional) feature the collector shall not be started at all until there is a cfg locally cfg to be merged or the opamp backend will send a cfg update to overwrite the default (noop). In this case also the error logs madness will not trigger - see my feature request #33680

[BinaryFissionGames]

I believe this ended up fixed in #34159