Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[receivercreator] Validate endpoint's configuration before starting receivers #35416

Merged

Conversation

ChrsMark
Copy link
Member

@ChrsMark ChrsMark commented Sep 25, 2024

Description:

The receivercreator should validate the runtime configurations before starting a receiver. This PR adds this validation.

Link to tracking Issue: Fixes #33145

Testing:

Using the following helm chart's values:

mode: daemonset

config:
  extensions:
    k8s_observer:
      auth_type: serviceAccount
      node: ${env:K8S_NODE_NAME}
      observe_nodes: true
  exporters:
    debug:
      verbosity: basic
  receivers:
    receiver_creator/3:
      watch_observers: [ k8s_observer ]
      receivers:
        apache:
          rule: type == "k8s.node"
          config:
            endpoint: "http://localhost:8080/server-status?manual"
  service:
    extensions: [health_check, k8s_observer]
    pipelines:
      metrics:
        receivers: [receiver_creator/3]
        processors: [batch]
        exporters: [debug]

Expect to see an error after the config's validation:

2024-09-25T10:46:56.778Z	error	receivercreator@v0.110.0/observerhandler.go:159	failed to start receiver	{"kind": "receiver", "name": "receiver_creator/3", "data_type": "metrics", "receiver": "apache", "error": "invalid runtime receiver config: receivers::apache: query must be 'auto': 'http://localhost:8080/server-status?manual'"}
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/receivercreator.(*observerHandler).OnAdd
	github.com/open-telemetry/opentelemetry-collector-contrib/receiver/receivercreator@v0.110.0/observerhandler.go:159
github.com/open-telemetry/opentelemetry-collector-contrib/extension/observer.(*EndpointsWatcher).updateAndNotifyOfEndpoints
	github.com/open-telemetry/opentelemetry-collector-contrib/extension/observer@v0.110.0/endpointswatcher.go:111

This stops the execution of the receiver which would lead to fetch errors (as described in the respective issue #33145):

2024-05-21T09:17:46.749Z	error	apachereceiver@v0.100.0/scraper.go:67	failed to fetch Apache Httpd stats	{"kind": "receiver", "name": "receiver_creator/3", "data_type": "metrics", "name": "apache//receiver_creator/3{endpoint=\"http://localhost:8080/server-status?manual\"}/k8s_observer/kind-control-plane-b75cdcf9-ee9d-4cb8-aa53-1ceb030d898d", "error": "Get \"http://localhost:8080/server-status?manual\": dial tcp [::1]:8080: connect: connection refused"}
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/apachereceiver.(*apacheScraper).scrape
	github.com/open-telemetry/opentelemetry-collector-contrib/receiver/apachereceiver@v0.100.0/scraper.go:67
go.opentelemetry.io/collector/receiver/scraperhelper.ScrapeFunc.Scrape
	go.opentelemetry.io/collector/receiver@v0.100.1-0.20240509190532-c555005fcc80/scraperhelper/scraper.go:20
go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).scrapeMetricsAndReport
	go.opentelemetry.io/collector/receiver@v0.100.1-0.20240509190532-c555005fcc80/scraperhelper/scrapercontroller.go:194
go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).startScraping.func1
	go.opentelemetry.io/collector/receiver@v0.100.1-0.20240509190532-c555005fcc80/scraperhelper/scrapercontroller.go:169
2024-05-21T09:17:46.750Z	error	scraperhelper/scrapercontroller.go:197	Error scraping metrics	{"kind": "receiver", "name": "receiver_creator/3", "data_type": "metrics", "name": "apache//receiver_creator/3{endpoint=\"http://localhost:8080/server-status?manual\"}/k8s_observer/kind-control-plane-b75cdcf9-ee9d-4cb8-aa53-1ceb030d898d", "error": "Get \"http://localhost:8080/server-status?manual\": dial tcp [::1]:8080: connect: connection refused", "scraper": "apache"}

Documentation:

…eceivers

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>
@ChrsMark
Copy link
Member Author

@dmitryax could you also have a look into this one🙏?

@dmitryax dmitryax merged commit 0f2fea6 into open-telemetry:main Sep 26, 2024
156 checks passed
@github-actions github-actions bot added this to the next release milestone Sep 26, 2024
jriguera pushed a commit to springernature/opentelemetry-collector-contrib that referenced this pull request Oct 4, 2024
…eceivers (open-telemetry#35416)

**Description:** 
The `receivercreator` should validate the runtime configurations before
starting a receiver. This PR adds this validation.

**Link to tracking Issue:** <Issue number if applicable> Fixes
open-telemetry#33145

**Testing:** 
Using the following helm chart's values:
```yaml
mode: daemonset

config:
  extensions:
    k8s_observer:
      auth_type: serviceAccount
      node: ${env:K8S_NODE_NAME}
      observe_nodes: true
  exporters:
    debug:
      verbosity: basic
  receivers:
    receiver_creator/3:
      watch_observers: [ k8s_observer ]
      receivers:
        apache:
          rule: type == "k8s.node"
          config:
            endpoint: "http://localhost:8080/server-status?manual"
  service:
    extensions: [health_check, k8s_observer]
    pipelines:
      metrics:
        receivers: [receiver_creator/3]
        processors: [batch]
        exporters: [debug]
```

Expect to see an error after the config's validation:

```console
2024-09-25T10:46:56.778Z	error	receivercreator@v0.110.0/observerhandler.go:159	failed to start receiver	{"kind": "receiver", "name": "receiver_creator/3", "data_type": "metrics", "receiver": "apache", "error": "invalid runtime receiver config: receivers::apache: query must be 'auto': 'http://localhost:8080/server-status?manual'"}
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/receivercreator.(*observerHandler).OnAdd
	github.com/open-telemetry/opentelemetry-collector-contrib/receiver/receivercreator@v0.110.0/observerhandler.go:159
github.com/open-telemetry/opentelemetry-collector-contrib/extension/observer.(*EndpointsWatcher).updateAndNotifyOfEndpoints
	github.com/open-telemetry/opentelemetry-collector-contrib/extension/observer@v0.110.0/endpointswatcher.go:111
```

This stops the execution of the receiver which would lead to fetch
errors (as described in the respective issue
open-telemetry#33145):

```console
2024-05-21T09:17:46.749Z	error	apachereceiver@v0.100.0/scraper.go:67	failed to fetch Apache Httpd stats	{"kind": "receiver", "name": "receiver_creator/3", "data_type": "metrics", "name": "apache//receiver_creator/3{endpoint=\"http://localhost:8080/server-status?manual\"}/k8s_observer/kind-control-plane-b75cdcf9-ee9d-4cb8-aa53-1ceb030d898d", "error": "Get \"http://localhost:8080/server-status?manual\": dial tcp [::1]:8080: connect: connection refused"}
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/apachereceiver.(*apacheScraper).scrape
	github.com/open-telemetry/opentelemetry-collector-contrib/receiver/apachereceiver@v0.100.0/scraper.go:67
go.opentelemetry.io/collector/receiver/scraperhelper.ScrapeFunc.Scrape
	go.opentelemetry.io/collector/receiver@v0.100.1-0.20240509190532-c555005fcc80/scraperhelper/scraper.go:20
go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).scrapeMetricsAndReport
	go.opentelemetry.io/collector/receiver@v0.100.1-0.20240509190532-c555005fcc80/scraperhelper/scrapercontroller.go:194
go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).startScraping.func1
	go.opentelemetry.io/collector/receiver@v0.100.1-0.20240509190532-c555005fcc80/scraperhelper/scrapercontroller.go:169
2024-05-21T09:17:46.750Z	error	scraperhelper/scrapercontroller.go:197	Error scraping metrics	{"kind": "receiver", "name": "receiver_creator/3", "data_type": "metrics", "name": "apache//receiver_creator/3{endpoint=\"http://localhost:8080/server-status?manual\"}/k8s_observer/kind-control-plane-b75cdcf9-ee9d-4cb8-aa53-1ceb030d898d", "error": "Get \"http://localhost:8080/server-status?manual\": dial tcp [::1]:8080: connect: connection refused", "scraper": "apache"}
```

**Documentation:** <Describe the documentation added.>

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

receivercreator does not honor target's config validation
3 participants