Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Prometheus receiver crashes on startup #2705

Closed
DanTulovsky opened this issue Mar 15, 2021 · 3 comments · Fixed by #3589
Closed

Prometheus receiver crashes on startup #2705

DanTulovsky opened this issue Mar 15, 2021 · 3 comments · Fixed by #3589
Labels
area:receiver bug Something isn't working

Comments

@DanTulovsky
Copy link

Describe the bug
otle-collector crashes on startup

Steps to reproduce
If possible, provide a recipe for reproducing the error.

What did you expect to see?
No crash on startup :)

What did you see instead?

2021-03-15T15:04:06.396Z info service/service.go:411 Starting OpenTelemetry Contrib Collector... {"Version": "v0.19.0", "GitHash": "63b2f339", "NumCPU": 2}
2021-03-15T15:04:06.396Z info service/service.go:255 Setting up own telemetry...
2021-03-15T15:04:06.398Z info service/telemetry.go:102 Serving Prometheus metrics {"address": ":8888", "level": 0, "service.instance.id": "173bdd06-d3bc-4903-ba90-eff33a3cc621"}
2021-03-15T15:04:06.399Z info service/service.go:292 Loading configuration...
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0xe8 pc=0x88a85c]

goroutine 1 [running]:
github.com/spf13/viper.(*Viper).AllKeys(0x0, 0x30, 0x6, 0xc000947301)
/home/circleci/go/pkg/mod/github.com/spf13/viper@v1.7.1/viper.go:1824 +0x9c
github.com/spf13/viper.(*Viper).AllSettings(0x0, 0x36e9b5a)
/home/circleci/go/pkg/mod/github.com/spf13/viper@v1.7.1/viper.go:1904 +0x51
go.opentelemetry.io/collector/receiver/prometheusreceiver.customUnmarshaler(0xc00088bd40, 0x313b880, 0xc000530840, 0xc0009474f0, 0xfbb7cc)
/home/circleci/go/pkg/mod/go.opentelemetry.io/collector@v0.19.0/receiver/prometheusreceiver/factory.go:70 +0x139
go.opentelemetry.io/collector/receiver/receiverhelper.(*factoryWithUnmarshaler).Unmarshal(0xc00012b150, 0xc00088bd40, 0x313b880, 0xc000530840, 0x7f984d43db48, 0xc00012b150)
/home/circleci/go/pkg/mod/go.opentelemetry.io/collector@v0.19.0/receiver/receiverhelper/factory.go:156 +0x4d
go.opentelemetry.io/collector/config.LoadReceiver(0xc00088bd40, 0xc00002baf6, 0xa, 0xc00002baf6, 0xa, 0x3e59100, 0xc00012b150, 0x0, 0xc0002f5560, 0x0, ...)
/home/circleci/go/pkg/mod/go.opentelemetry.io/collector@v0.19.0/config/config.go:319 +0x11f
go.opentelemetry.io/collector/config.loadReceivers(0xc00086fa40, 0xc000569770, 0x9, 0xc00086fa40, 0x0)
/home/circleci/go/pkg/mod/go.opentelemetry.io/collector@v0.19.0/config/config.go:347 +0x1df
go.opentelemetry.io/collector/config.Load(0xc000287b00, 0xc000569770, 0xc000504300, 0xc000569a70, 0xc000569740, 0x52, 0x124, 0x5a3ea88)
/home/circleci/go/pkg/mod/go.opentelemetry.io/collector@v0.19.0/config/config.go:159 +0x258
go.opentelemetry.io/collector/service.FileLoaderConfigFactory(0xc000287b00, 0xc0006df340, 0xc000569770, 0xc000504300, 0xc000569a70, 0xc000569740, 0xc00030ff20, 0x0, 0x0)
/home/circleci/go/pkg/mod/go.opentelemetry.io/collector@v0.19.0/service/service.go:138 +0x25a
go.opentelemetry.io/collector/service.(*Application).setupConfigurationComponents(0xc0002f0180, 0x3e41d80, 0xc000116000, 0x382a9b0, 0x0, 0x3)
/home/circleci/go/pkg/mod/go.opentelemetry.io/collector@v0.19.0/service/service.go:293 +0xf8
go.opentelemetry.io/collector/service.(*Application).execute(0xc0002f0180, 0x3e41d80, 0xc000116000, 0x382a9b0, 0x0, 0x0)
/home/circleci/go/pkg/mod/go.opentelemetry.io/collector@v0.19.0/service/service.go:429 +0x4d7

What version did you use?
Version: v0.19.0

What config did you use?
Config:

receivers:
  otlp:
    protocols:
      grpc:
      http:
  opencensus:
  jaeger:
    protocols:
      grpc:
      thrift_http:
      thrift_compact:
      thrift_binary:
    remote_sampling:
      strategy_file: "/etc/jaeger/jaeger-sampling-config.json"
      # Doesn't work
      # fetch_endpoint: "jaeger-collector.observability:5778"
  zipkin:
  prometheus:
    config:
      - job_name: ambassador/ambassador/0
        honor_labels: false
        kubernetes_sd_configs:
        - role: endpoints
          namespaces:
            names:
            - ambassador
        metrics_path: /metrics
        relabel_configs:
        - action: keep
          source_labels:
          - __meta_kubernetes_service_label_service
          regex: ambassador-admin
        - action: keep
          source_labels:
          - __meta_kubernetes_endpoint_port_name
          regex: ambassador-admin
        - source_labels:
          - __meta_kubernetes_endpoint_address_target_kind
          - __meta_kubernetes_endpoint_address_target_name
          separator: ;
          regex: Node;(.*)
          replacement: ${1}
          target_label: node
        - source_labels:
          - __meta_kubernetes_endpoint_address_target_kind
          - __meta_kubernetes_endpoint_address_target_name
          separator: ;
          regex: Pod;(.*)
          replacement: ${1}
          target_label: pod
        - source_labels:
          - __meta_kubernetes_namespace
          target_label: namespace
        - source_labels:
          - __meta_kubernetes_service_name
          target_label: service
        - source_labels:
          - __meta_kubernetes_pod_name
          target_label: pod
        - source_labels:
          - __meta_kubernetes_pod_container_name
          target_label: container
        - source_labels:
          - __meta_kubernetes_service_name
          target_label: job
          replacement: ${1}
        - target_label: endpoint
          replacement: ambassador-admin
        - source_labels:
          - __address__
          target_label: __tmp_hash
          modulus: 1
          action: hashmod
        - source_labels:
          - __tmp_hash
          regex: 0
          action: keep
      - job_name: auth/keycloak/0
        honor_labels: false
        kubernetes_sd_configs:
        - role: endpoints
          namespaces:
            names:
            - auth
        scrape_interval: 30s
        relabel_configs:
        - action: keep
          source_labels:
          - __meta_kubernetes_service_label_app_kubernetes_io_component
          regex: metrics
        - action: keep
          source_labels:
          - __meta_kubernetes_service_label_app_kubernetes_io_instance
          regex: keycloak
        - action: keep
          source_labels:
          - __meta_kubernetes_service_label_app_kubernetes_io_name
          regex: keycloak
        - action: keep
          source_labels:
          - __meta_kubernetes_endpoint_port_name
          regex: http-management
        - source_labels:
          - __meta_kubernetes_endpoint_address_target_kind
          - __meta_kubernetes_endpoint_address_target_name
          separator: ;
          regex: Node;(.*)
          replacement: ${1}
          target_label: node
        - source_labels:
          - __meta_kubernetes_endpoint_address_target_kind
          - __meta_kubernetes_endpoint_address_target_name
          separator: ;
          regex: Pod;(.*)
          replacement: ${1}
          target_label: pod
        - source_labels:
          - __meta_kubernetes_namespace
          target_label: namespace
        - source_labels:
          - __meta_kubernetes_service_name
          target_label: service
        - source_labels:
          - __meta_kubernetes_pod_name
          target_label: pod
        - source_labels:
          - __meta_kubernetes_pod_container_name
          target_label: container
        - source_labels:
          - __meta_kubernetes_service_name
          target_label: job
          replacement: ${1}
        - target_label: endpoint
          replacement: http-management
        - source_labels:
          - __address__
          target_label: __tmp_hash
          modulus: 1
          action: hashmod
        - source_labels:
          - __tmp_hash
          regex: 0
          action: keep
      - job_name: monitoring/grafana0/0
        honor_labels: true
        kubernetes_sd_configs:
        - role: endpoints
          namespaces:
            names:
            - monitoring
        scrape_interval: 1m
        scrape_timeout: 30s
        metrics_path: /metrics
        scheme: http
        relabel_configs:
        - action: keep
          source_labels:
          - __meta_kubernetes_service_label_app_kubernetes_io_instance
          regex: grafana0
        - action: keep
          source_labels:
          - __meta_kubernetes_service_label_app_kubernetes_io_name
          regex: grafana
        - action: keep
          source_labels:
          - __meta_kubernetes_endpoint_port_name
          regex: service
        - source_labels:
          - __meta_kubernetes_endpoint_address_target_kind
          - __meta_kubernetes_endpoint_address_target_name
          separator: ;
          regex: Node;(.*)
          replacement: ${1}
          target_label: node
        - source_labels:
          - __meta_kubernetes_endpoint_address_target_kind
          - __meta_kubernetes_endpoint_address_target_name
          separator: ;
          regex: Pod;(.*)
          replacement: ${1}
          target_label: pod
        - source_labels:
          - __meta_kubernetes_namespace
          target_label: namespace
        - source_labels:
          - __meta_kubernetes_service_name
          target_label: service
        - source_labels:
          - __meta_kubernetes_pod_name
          target_label: pod
        - source_labels:
          - __meta_kubernetes_pod_container_name
          target_label: container
        - source_labels:
          - __meta_kubernetes_service_name
          target_label: job
          replacement: ${1}
        - source_labels:
          - __meta_kubernetes_service_label_grafana0
          target_label: job
          regex: (.+)
          replacement: ${1}
        - target_label: endpoint
          replacement: service
        - source_labels:
          - __address__
          target_label: __tmp_hash
          modulus: 1
          action: hashmod
        - source_labels:
          - __tmp_hash
          regex: 0
          action: keep
      - job_name: monitoring/prom0-kube-prometheus-apiserver/0
        honor_labels: false
        kubernetes_sd_configs:
        - role: endpoints
          namespaces:
            names:
            - default
        scheme: https
        tls_config:
          insecure_skip_verify: true
          server_name: kubernetes
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        relabel_configs:
        - action: keep
          source_labels:
          - __meta_kubernetes_service_label_component
          regex: apiserver
        - action: keep
          source_labels:
          - __meta_kubernetes_service_label_provider
          regex: kubernetes
        - action: keep
          source_labels:
          - __meta_kubernetes_endpoint_port_name
          regex: https
        - source_labels:
          - __meta_kubernetes_endpoint_address_target_kind
          - __meta_kubernetes_endpoint_address_target_name
          separator: ;
          regex: Node;(.*)
          replacement: ${1}
          target_label: node
        - source_labels:
          - __meta_kubernetes_endpoint_address_target_kind
          - __meta_kubernetes_endpoint_address_target_name
          separator: ;
          regex: Pod;(.*)
          replacement: ${1}
          target_label: pod
        - source_labels:
          - __meta_kubernetes_namespace
          target_label: namespace
        - source_labels:
          - __meta_kubernetes_service_name
          target_label: service
        - source_labels:
          - __meta_kubernetes_pod_name
          target_label: pod
        - source_labels:
          - __meta_kubernetes_pod_container_name
          target_label: container
        - source_labels:
          - __meta_kubernetes_service_name
          target_label: job
          replacement: ${1}
        - source_labels:
          - __meta_kubernetes_service_label_component
          target_label: job
          regex: (.+)
          replacement: ${1}
        - target_label: endpoint
          replacement: https
        - source_labels:
          - __address__
          target_label: __tmp_hash
          modulus: 1
          action: hashmod
        - source_labels:
          - __tmp_hash
          regex: 0
          action: keep
      - job_name: monitoring/prom0-kube-prometheus-coredns/0
        honor_labels: false
        kubernetes_sd_configs:
        - role: endpoints
          namespaces:
            names:
            - kube-system
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        relabel_configs:
        - action: keep
          source_labels:
          - __meta_kubernetes_service_label_app_kubernetes_io_component
          regex: prom0-kube-prometheus-coredns
        - action: keep
          source_labels:
          - __meta_kubernetes_endpoint_port_name
          regex: http-metrics
        - source_labels:
          - __meta_kubernetes_endpoint_address_target_kind
          - __meta_kubernetes_endpoint_address_target_name
          separator: ;
          regex: Node;(.*)
          replacement: ${1}
          target_label: node
        - source_labels:
          - __meta_kubernetes_endpoint_address_target_kind
          - __meta_kubernetes_endpoint_address_target_name
          separator: ;
          regex: Pod;(.*)
          replacement: ${1}
          target_label: pod
        - source_labels:
          - __meta_kubernetes_namespace
          target_label: namespace
        - source_labels:
          - __meta_kubernetes_service_name
          target_label: service
        - source_labels:
          - __meta_kubernetes_pod_name
          target_label: pod
        - source_labels:
          - __meta_kubernetes_pod_container_name
          target_label: container
        - source_labels:
          - __meta_kubernetes_service_name
          target_label: job
          replacement: ${1}
        - source_labels:
          - __meta_kubernetes_service_label_k8s_app
          target_label: job
          regex: (.+)
          replacement: ${1}
        - target_label: endpoint
          replacement: http-metrics
        - source_labels:
          - __address__
          target_label: __tmp_hash
          modulus: 1
          action: hashmod
        - source_labels:
          - __tmp_hash
          regex: 0
          action: keep
      - job_name: monitoring/prom0-kube-prometheus-kube-proxy/0
        honor_labels: false
        kubernetes_sd_configs:
        - role: endpoints
          namespaces:
            names:
            - kube-system
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        relabel_configs:
        - action: keep
          source_labels:
          - __meta_kubernetes_service_label_app_kubernetes_io_component
          regex: kube-proxy
        - action: keep
          source_labels:
          - __meta_kubernetes_endpoint_port_name
          regex: http-metrics
        - source_labels:
          - __meta_kubernetes_endpoint_address_target_kind
          - __meta_kubernetes_endpoint_address_target_name
          separator: ;
          regex: Node;(.*)
          replacement: ${1}
          target_label: node
        - source_labels:
          - __meta_kubernetes_endpoint_address_target_kind
          - __meta_kubernetes_endpoint_address_target_name
          separator: ;
          regex: Pod;(.*)
          replacement: ${1}
          target_label: pod
        - source_labels:
          - __meta_kubernetes_namespace
          target_label: namespace
        - source_labels:
          - __meta_kubernetes_service_name
          target_label: service
        - source_labels:
          - __meta_kubernetes_pod_name
          target_label: pod
        - source_labels:
          - __meta_kubernetes_pod_container_name
          target_label: container
        - source_labels:
          - __meta_kubernetes_service_name
          target_label: job
          replacement: ${1}
        - source_labels:
          - __meta_kubernetes_service_label_k8s_app
          target_label: job
          regex: (.+)
          replacement: ${1}
        - target_label: endpoint
          replacement: http-metrics
        - source_labels:
          - __address__
          target_label: __tmp_hash
          modulus: 1
          action: hashmod
        - source_labels:
          - __tmp_hash
          regex: 0
          action: keep
      - job_name: monitoring/prom0-kube-prometheus-kube-scheduler/0
        honor_labels: false
        kubernetes_sd_configs:
        - role: endpoints
          namespaces:
            names:
            - kube-system
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        relabel_configs:
        - action: keep
          source_labels:
          - __meta_kubernetes_service_label_app_kubernetes_io_component
          regex: prom0-kube-prometheus-kube-scheduler
        - action: keep
          source_labels:
          - __meta_kubernetes_endpoint_port_name
          regex: http-metrics
        - source_labels:
          - __meta_kubernetes_endpoint_address_target_kind
          - __meta_kubernetes_endpoint_address_target_name
          separator: ;
          regex: Node;(.*)
          replacement: ${1}
          target_label: node
        - source_labels:
          - __meta_kubernetes_endpoint_address_target_kind
          - __meta_kubernetes_endpoint_address_target_name
          separator: ;
          regex: Pod;(.*)
          replacement: ${1}
          target_label: pod
        - source_labels:
          - __meta_kubernetes_namespace
          target_label: namespace
        - source_labels:
          - __meta_kubernetes_service_name
          target_label: service
        - source_labels:
          - __meta_kubernetes_pod_name
          target_label: pod
        - source_labels:
          - __meta_kubernetes_pod_container_name
          target_label: container
        - source_labels:
          - __meta_kubernetes_service_name
          target_label: job
          replacement: ${1}
        - source_labels:
          - __meta_kubernetes_service_label_component
          target_label: job
          regex: (.+)
          replacement: ${1}
        - target_label: endpoint
          replacement: http-metrics
        - source_labels:
          - __address__
          target_label: __tmp_hash
          modulus: 1
          action: hashmod
        - source_labels:
          - __tmp_hash
          regex: 0
          action: keep
      - job_name: monitoring/prom0-kube-prometheus-kubelet/0
        honor_labels: true
        kubernetes_sd_configs:
        - role: endpoints
          namespaces:
            names:
            - kube-system
        scheme: https
        tls_config:
          insecure_skip_verify: true
          server_name: kubernetes
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        relabel_configs:
        - action: keep
          source_labels:
          - __meta_kubernetes_service_label_k8s_app
          regex: kubelet
        - action: keep
          source_labels:
          - __meta_kubernetes_endpoint_port_name
          regex: https-metrics
        - source_labels:
          - __meta_kubernetes_endpoint_address_target_kind
          - __meta_kubernetes_endpoint_address_target_name
          separator: ;
          regex: Node;(.*)
          replacement: ${1}
          target_label: node
        - source_labels:
          - __meta_kubernetes_endpoint_address_target_kind
          - __meta_kubernetes_endpoint_address_target_name
          separator: ;
          regex: Pod;(.*)
          replacement: ${1}
          target_label: pod
        - source_labels:
          - __meta_kubernetes_namespace
          target_label: namespace
        - source_labels:
          - __meta_kubernetes_service_name
          target_label: service
        - source_labels:
          - __meta_kubernetes_pod_name
          target_label: pod
        - source_labels:
          - __meta_kubernetes_pod_container_name
          target_label: container
        - source_labels:
          - __meta_kubernetes_service_name
          target_label: job
          replacement: ${1}
        - source_labels:
          - __meta_kubernetes_service_label_k8s_app
          target_label: job
          regex: (.+)
          replacement: ${1}
        - target_label: endpoint
          replacement: https-metrics
        - source_labels:
          - __address__
          target_label: __tmp_hash
          modulus: 1
          action: hashmod
        - source_labels:
          - __tmp_hash
          regex: 0
          action: keep
        metric_relabel_configs:
        - source_labels:
          - node
          target_label: instance
          action: replace
      - job_name: monitoring/prom0-kube-prometheus-kubelet/1
        honor_labels: true
        kubernetes_sd_configs:
        - role: endpoints
          namespaces:
            names:
            - kube-system
        metrics_path: /metrics/cadvisor
        scheme: https
        tls_config:
          insecure_skip_verify: true
          server_name: kubernetes
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        relabel_configs:
        - action: keep
          source_labels:
          - __meta_kubernetes_service_label_k8s_app
          regex: kubelet
        - action: keep
          source_labels:
          - __meta_kubernetes_endpoint_port_name
          regex: https-metrics
        - source_labels:
          - __meta_kubernetes_endpoint_address_target_kind
          - __meta_kubernetes_endpoint_address_target_name
          separator: ;
          regex: Node;(.*)
          replacement: ${1}
          target_label: node
        - source_labels:
          - __meta_kubernetes_endpoint_address_target_kind
          - __meta_kubernetes_endpoint_address_target_name
          separator: ;
          regex: Pod;(.*)
          replacement: ${1}
          target_label: pod
        - source_labels:
          - __meta_kubernetes_namespace
          target_label: namespace
        - source_labels:
          - __meta_kubernetes_service_name
          target_label: service
        - source_labels:
          - __meta_kubernetes_pod_name
          target_label: pod
        - source_labels:
          - __meta_kubernetes_pod_container_name
          target_label: container
        - source_labels:
          - __meta_kubernetes_service_name
          target_label: job
          replacement: ${1}
        - source_labels:
          - __meta_kubernetes_service_label_k8s_app
          target_label: job
          regex: (.+)
          replacement: ${1}
        - target_label: endpoint
          replacement: https-metrics
        - source_labels:
          - __address__
          target_label: __tmp_hash
          modulus: 1
          action: hashmod
        - source_labels:
          - __tmp_hash
          regex: 0
          action: keep
      - job_name: monitoring/prom0-kube-prometheus-operator/0
        honor_labels: true
        kubernetes_sd_configs:
        - role: endpoints
          namespaces:
            names:
            - monitoring
        relabel_configs:
        - action: keep
          source_labels:
          - __meta_kubernetes_service_label_app_kubernetes_io_component
          regex: operator
        - action: keep
          source_labels:
          - __meta_kubernetes_service_label_app_kubernetes_io_instance
          regex: prom0
        - action: keep
          source_labels:
          - __meta_kubernetes_service_label_app_kubernetes_io_name
          regex: kube-prometheus
        - action: keep
          source_labels:
          - __meta_kubernetes_endpoint_port_name
          regex: http
        - source_labels:
          - __meta_kubernetes_endpoint_address_target_kind
          - __meta_kubernetes_endpoint_address_target_name
          separator: ;
          regex: Node;(.*)
          replacement: ${1}
          target_label: node
        - source_labels:
          - __meta_kubernetes_endpoint_address_target_kind
          - __meta_kubernetes_endpoint_address_target_name
          separator: ;
          regex: Pod;(.*)
          replacement: ${1}
          target_label: pod
        - source_labels:
          - __meta_kubernetes_namespace
          target_label: namespace
        - source_labels:
          - __meta_kubernetes_service_name
          target_label: service
        - source_labels:
          - __meta_kubernetes_pod_name
          target_label: pod
        - source_labels:
          - __meta_kubernetes_pod_container_name
          target_label: container
        - source_labels:
          - __meta_kubernetes_service_name
          target_label: job
          replacement: ${1}
        - target_label: endpoint
          replacement: http
        - source_labels:
          - __address__
          target_label: __tmp_hash
          modulus: 1
          action: hashmod
        - source_labels:
          - __tmp_hash
          regex: 0
          action: keep
      - job_name: monitoring/prom0-kube-prometheus-prometheus/0
        honor_labels: false
        kubernetes_sd_configs:
        - role: endpoints
          namespaces:
            names:
            - monitoring
        metrics_path: /metrics
        relabel_configs:
        - action: keep
          source_labels:
          - __meta_kubernetes_service_label_app_kubernetes_io_component
          regex: prometheus
        - action: keep
          source_labels:
          - __meta_kubernetes_service_label_app_kubernetes_io_instance
          regex: prom0
        - action: keep
          source_labels:
          - __meta_kubernetes_service_label_app_kubernetes_io_name
          regex: kube-prometheus
        - action: keep
          source_labels:
          - __meta_kubernetes_endpoint_port_name
          regex: http
        - source_labels:
          - __meta_kubernetes_endpoint_address_target_kind
          - __meta_kubernetes_endpoint_address_target_name
          separator: ;
          regex: Node;(.*)
          replacement: ${1}
          target_label: node
        - source_labels:
          - __meta_kubernetes_endpoint_address_target_kind
          - __meta_kubernetes_endpoint_address_target_name
          separator: ;
          regex: Pod;(.*)
          replacement: ${1}
          target_label: pod
        - source_labels:
          - __meta_kubernetes_namespace
          target_label: namespace
        - source_labels:
          - __meta_kubernetes_service_name
          target_label: service
        - source_labels:
          - __meta_kubernetes_pod_name
          target_label: pod
        - source_labels:
          - __meta_kubernetes_pod_container_name
          target_label: container
        - source_labels:
          - __meta_kubernetes_service_name
          target_label: job
          replacement: ${1}
        - target_label: endpoint
          replacement: http
        - source_labels:
          - __address__
          target_label: __tmp_hash
          modulus: 1
          action: hashmod
        - source_labels:
          - __tmp_hash
          regex: 0
          action: keep
      - job_name: monitoring/prom0-node-exporter/0
        honor_labels: false
        kubernetes_sd_configs:
        - role: endpoints
          namespaces:
            names:
            - monitoring
        relabel_configs:
        - action: keep
          source_labels:
          - __meta_kubernetes_service_label_app_kubernetes_io_instance
          regex: prom0
        - action: keep
          source_labels:
          - __meta_kubernetes_service_label_app_kubernetes_io_name
          regex: node-exporter
        - action: keep
          source_labels:
          - __meta_kubernetes_endpoint_port_name
          regex: metrics
        - source_labels:
          - __meta_kubernetes_endpoint_address_target_kind
          - __meta_kubernetes_endpoint_address_target_name
          separator: ;
          regex: Node;(.*)
          replacement: ${1}
          target_label: node
        - source_labels:
          - __meta_kubernetes_endpoint_address_target_kind
          - __meta_kubernetes_endpoint_address_target_name
          separator: ;
          regex: Pod;(.*)
          replacement: ${1}
          target_label: pod
        - source_labels:
          - __meta_kubernetes_namespace
          target_label: namespace
        - source_labels:
          - __meta_kubernetes_service_name
          target_label: service
        - source_labels:
          - __meta_kubernetes_pod_name
          target_label: pod
        - source_labels:
          - __meta_kubernetes_pod_container_name
          target_label: container
        - source_labels:
          - __meta_kubernetes_service_name
          target_label: job
          replacement: ${1}
        - source_labels:
          - __meta_kubernetes_service_label_jobLabel
          target_label: job
          regex: (.+)
          replacement: ${1}
        - target_label: endpoint
          replacement: metrics
        - source_labels:
          - __meta_kubernetes_pod_node_name
          target_label: kubernetes_node
          regex: (.*)
          replacement: $1
          action: replace
        - source_labels:
          - __address__
          target_label: __tmp_hash
          modulus: 1
          action: hashmod
        - source_labels:
          - __tmp_hash
          regex: 0
          action: keep
      - job_name: observability/otel-collector-monitor/0
        honor_labels: false
        kubernetes_sd_configs:
        - role: endpoints
        relabel_configs:
        - action: keep
          source_labels:
          - __meta_kubernetes_service_label_service
          regex: otel-collector
        - action: keep
          source_labels:
          - __meta_kubernetes_endpoint_port_name
          regex: metrics
        - source_labels:
          - __meta_kubernetes_endpoint_address_target_kind
          - __meta_kubernetes_endpoint_address_target_name
          separator: ;
          regex: Node;(.*)
          replacement: ${1}
          target_label: node
        - source_labels:
          - __meta_kubernetes_endpoint_address_target_kind
          - __meta_kubernetes_endpoint_address_target_name
          separator: ;
          regex: Pod;(.*)
          replacement: ${1}
          target_label: pod
        - source_labels:
          - __meta_kubernetes_namespace
          target_label: namespace
        - source_labels:
          - __meta_kubernetes_service_name
          target_label: service
        - source_labels:
          - __meta_kubernetes_pod_name
          target_label: pod
        - source_labels:
          - __meta_kubernetes_pod_container_name
          target_label: container
        - source_labels:
          - __meta_kubernetes_service_name
          target_label: job
          replacement: ${1}
        - target_label: endpoint
          replacement: metrics
        - source_labels:
          - __address__
          target_label: __tmp_hash
          modulus: 1
          action: hashmod
        - source_labels:
          - __tmp_hash
          regex: 0
          action: keep
      - job_name: web/static-web-monitor/0
        honor_labels: false
        kubernetes_sd_configs:
        - role: endpoints
        relabel_configs:
        - action: keep
          source_labels:
          - __meta_kubernetes_service_label_service
          regex: static-web-frontend
        - action: keep
          source_labels:
          - __meta_kubernetes_endpoint_port_name
          regex: http
        - source_labels:
          - __meta_kubernetes_endpoint_address_target_kind
          - __meta_kubernetes_endpoint_address_target_name
          separator: ;
          regex: Node;(.*)
          replacement: ${1}
          target_label: node
        - source_labels:
          - __meta_kubernetes_endpoint_address_target_kind
          - __meta_kubernetes_endpoint_address_target_name
          separator: ;
          regex: Pod;(.*)
          replacement: ${1}
          target_label: pod
        - source_labels:
          - __meta_kubernetes_namespace
          target_label: namespace
        - source_labels:
          - __meta_kubernetes_service_name
          target_label: service
        - source_labels:
          - __meta_kubernetes_pod_name
          target_label: pod
        - source_labels:
          - __meta_kubernetes_pod_container_name
          target_label: container
        - source_labels:
          - __meta_kubernetes_service_name
          target_label: job
          replacement: ${1}
        - target_label: endpoint
          replacement: http
        - source_labels:
          - __address__
          target_label: __tmp_hash
          modulus: 1
          action: hashmod
        - source_labels:
          - __tmp_hash
          regex: 0
          action: keep
  # k8s_cluster:
  # auth_type: "serviceAccount"
  # collection_interval: 10s
  # node_conditions_to_report: [Ready, MemoryPressure, DiskPressure, PIDPressure, NetworkUnavailable]
processors:
  batch:
  memory_limiter:
    # Same as --mem-ballast-size-mib CLI argument
    # ballast_size_mib: 128
    # 80% of maximum memory up to 2G
    limit_mib: 450
    # 25% of limit up to 2G
    spike_limit_mib: 128
    check_interval: 5s
  k8s_tagger:
    passthrough: true
extensions:
  health_check: {}
  # https://github.com/open-telemetry/opentelemetry-specification/blob/main/experimental/trace/zpages.md
  zpages: {}
  pprof: {}
exporters:
  otlp/lightstep:
    endpoint: "ingest.lightstep.com:443"
    headers: {"lightstep-access-token": ${lightstepAccessToken}}
  # jaeger_thrift:
  #   url: "http://jaeger-collector.observability:14268/api/traces"
  logging:
    loglevel: info
  kafka:
    protocol_version: 2.0.0
    brokers:
      - "kafka0-headless.kafka:9092"
service:
  extensions: [health_check, zpages, pprof]
  pipelines:
    metrics:
      receivers: [otlp, opencensus]
      processors: [memory_limiter, batch]
      exporters: [otlp/lightstep]
    traces/1:
      receivers: [otlp, zipkin, jaeger]
      processors: [memory_limiter, batch, k8s_tagger]
      exporters: [otlp/lightstep]

Environment
OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): GKE

Additional context
The crash started when I added the prometheus receiver, everything else worked fine before that.
@DanTulovsky DanTulovsky added the bug Something isn't working label Mar 15, 2021
@yeya24
Copy link
Contributor

yeya24 commented Mar 16, 2021 •
edited
Loading

This is a configuration problem.
You need to add scrape_configs: in your prometheus config section.

  prometheus:
    config:
      scrape_configs:
      - job_name: ambassador/ambassador/0
        honor_labels: false
      ...

@DanTulovsky
Copy link
Author

oh.. duh, sorry. But perhaps the binary shouldn't crash on an invalid config? Or does it do that on purpose?

@bogdandrutu
Copy link
Member

It does that on purpose, we are working to add a dry-run flag just to validate config soon.

@DanTulovsky DanTulovsky changed the title Prometheus receiver crashes on staertup Prometheus receiver crashes on startup Mar 24, 2021
@rakyll rakyll mentioned this issue Jul 9, 2021
Merged
bogdandrutu pushed a commit that referenced this issue Jul 13, 2021
@rakyll
Validate the Prometheus configuration (#3589)
ae42627 
* Validate the Prometheus config

Validate against nil configuration and configuration with no scrape_configs.

Fixes #2705.

* Allow initialization of recievers without any configuration
Troels51 pushed a commit to Troels51/opentelemetry-collector that referenced this issue Jul 5, 2024
@dependabot
Bump docker/build-push-action from 5 to 6 (open-telemetry#2705)
595801b 
Bumps [docker/build-push-action](https://github.com/docker/build-push-action) from 5 to 6.
- [Release notes](https://github.com/docker/build-push-action/releases)
- [Commits](docker/build-push-action@v5...v6)

---
updated-dependencies:
- dependency-name: docker/build-push-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area:receiver bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Validate the Prometheus configuration
4 participants
@DanTulovsky @bogdandrutu @alolita @yeya24