[receiver/receivercreator] Support for probing K8s services

[a-thaler]

Component(s)

receiver/receivercreator
extension/observer

Is your feature request related to a problem? Please describe.

I used the typical setup of blackbox-exporter and prometheus to probe the availability of k8s services and make the results available as metrics. Hereby, the discovery of services by prometheus annotations is an elegant approach.

With the availability of the httpcheck receiver the setup can be turned into an otel-collector based setup. However, the configuration is static and does not support any discovery capabilities.

The receivercreator reciever seems the perfect match for that scenario by configuring the httpcheck dynamically dependent on the discovered service endpoints.

Describe the solution you'd like

Add support to the k8s_observer to watch k8s services, so that a new rule type k8s.service is available in the receivercreator reciever which allows the dynamic configuration of the httpcheck receiver.

An example configuration might look like this:

  extensions:
    k8s_observer:
      observe_nodes: false
      observe_pods: false
      observe_services: true

  receivers:
    receiver_creator:
      watch_observers: [k8s_observer]
      receivers:
        httpcheck:
          rule: type == "k8s.service" && annotations["prometheus.io/probe"] == "true"
          config:
            targets:
              - endpoint: 'http://`endpoint`:9090/health'
                method: 'GET'
            collection_interval: 10s

  exporters:
    debug:
      verbosity: detailed

  service:
    extensions:
      - k8s_observer
    pipelines:
      metrics:
        receivers:
        - receiver_creator
        exporters:
        - debug

A more advanced config could look like this:

  receivers:
    receiver_creator:
      watch_observers: [k8s_observer]
      receivers:
        httpcheck:
          rule: type == "k8s.service" && annotations["prometheus.io/probe"] == "true"
          config:
            targets:
              - endpoint: '`"prometheus.io/scheme" in annotations ? annotations["prometheus.io/scheme"] : http`://`endpoint`:`"prometheus.io/port" in annotations ? annotations["prometheus.io/port"] : 9090``"prometheus.io/path" in annotations ? annotations["prometheus.io/path"] : "/health"`'
                method: '`"prometheus.io/method" in annotations ? annotations["prometheus.io/method"] : GET`'
            collection_interval: '`pod.annotations["collection_interval"]`'

Describe alternatives you've considered

As the httpcheck receiver does not support any discovery, the only alternative is to stick to the blackbox-exporter and use the prometheusreceiver with a target_allocator instead which is way more complex and requires the blackbox-exporter.

Additional context

No response

[github-actions]

Pinging code owners:

• receiver/receivercreator: @rmfitzpatrick

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[a-thaler]

I created a first PR for it: #29022
Additional points I see which could be addressed:

Service Port Discovery
Analogue to the pod discovery, also the service ports could be transformed into individual endpoints. Here I'm in doubt if the existing port type should be re-used and just extended with optional service metadata or if a new port type needs to be introduced. Extending the existing type will require to check in the config if a port belongs to a pod or a service. So probably a new type like k8s.service.port should be introduced for that to clearly distinguish the two kind of ports.

Resource Attributes
The PR proposes to add the namespace by default as resource attribute. Unfortunately there is nothing available in the semantic conventions to reflect a k8s service as the source of the data.

Service attributes
In the PR I used the internal DNS name of a service as endpoint. If the user prefers to use the IP, it is available via the cluster_ip attribute. Additional I could extend the PR to also list the external host name in case of a LoadBalancer service, so that an external probing could be done as well. I currently don't se other attributes of interest.

[JaredTan95]

@a-thaler Hi, As I kown, blackbox-exporter not only do http probe, but also do tcp, ssh,imap etc. so, you cared about just http probe?

[JaredTan95]

@a-thaler Hi, As I kown, blackbox-exporter not only do http probe, but also do tcp, ssh,imap etc. so, you cared about just http probe?

refs #29009

[a-thaler]

Hi @JaredTan95, I currently care about HTTP and like to leverage the httpcheck receiver, but the approach outlined here is agnostic to the protocol as it determines the endpoint only and provides you the service context. It then depends on the actual nested receiver config. So a potential receiver resulting out of the linked feature request should be usable as well.

[JaredTan95]

but the approach outlined here is agnostic to the protocol as it determines the endpoint only and provides you the service context. It then depends on the actual nested receiver config. So a potential receiver resulting out of the linked feature request should be usable as well.

make sense

[a-thaler]

Feature got merged as proposed with #29022. Further improvements will be handled in individual issues.