Metrics Exporter not using configured URI

[cobar79]

Describe the bug
Apologies if this not the correct place, but I can't find any examples pushing metrics using OTEL collector and Mimir backend.
Problem set is pushing metrics from Spring Boot 3 to Collector and exporting to Mimir backend is failing for connection refused. It appears the OTLP URI is not being changed when attempting to export. Traces and Logs are exporting correctly.

Following Grafana Mimir collector configuration
First attempt to export using "prometheusremotewrite" export showed no errors and nothing was exported.
Second attempt to export to "otlphttp" endpoint revealed a

Post \"http://localhost:9009/otlp/v1/metrics\": dial tcp 127.0.0.1:9009: connect: connection refused

Noting the endpoint is the receiver value "http://localhost:4318/v1/metrics" via spring boot actuator value. The export should have been "http://localhost:9009/otlp" not "http://localhost:9009/otlp/v1/metrics"

Do I need to do something in the processor to remove the v1/metrics?

Environment:
Spring Boot 3.2.1
Docker 4.15.0
Patterned after Play With Mirmir and adapted to OTEL Collector.

Steps to reproduce

What did you expect to see?
Metrics showing up in Dashboard. Metrics persisted in Minio. Prometheus (Mimir) explore to contain metrics to search. Prometheus should be scraping Minio DB.

What did you see instead?
Nothing in minio database console.
Nothing in Mimir Write dashboard.
Nothing in the Mimir explore dropdowns.

What version did you use?

What config did you use?
compose:

version: '3'

services:
  sonarqube:
    container_name: sonarqube
    image: sonarqube:latest
    ports:
      - 9000:9000
      - 9092:9092

  auth-server:
    container_name: auth-server
    image: quay.io/keycloak/keycloak:latest
    ports:
      - 9080:8080
    environment:
      KEYCLOAK_HOSTNAME: osint-keycloak
      KEYCLOAK_ADMIN: admin
      KEYCLOAK_ADMIN_PASSWORD: admin
    volumes:
      - ${OSI_DOCKER_ROOT}\realm:/opt/keycloak/data/import
    command: -v start-dev --import-realm

  rabbit:
    container_name: RogerRabbit
    image: rabbitmq:3-management
    ports:
      - 15672:15672
      - 5672:5672

  loki:
    container_name: loki
    image: grafana/loki:latest
    command: [ "-config.file=/etc/loki/local-config.yaml" ]
    ports:
      - "3100:3100"

  tempo:
    container_name: tempo
    image: grafana/tempo:latest
    command: [ "-config.file=/etc/tempo.yml" ]
    environment:
      - OTEL_SPAN_EVENT_COUNT_LIMIT=3000
    volumes:
      - ${OSI_DOCKER_ROOT}\tempo\tempo.yml:/etc/tempo.yml:ro
      - ${OSI_DOCKER_ROOT}\tempo\tempo-data:/tmp/tempo
    ports:
      - "3200:3200"  # Tempo see tempo.yml http_listen_port
      - "4317"  # otlp grpc

########### MIMIR:: PROMETHEUS DATASOURCE #######
  #Configure Mimir to use Minio as S3 object storage backend.
  #Change default port from 9000 to 9900 since sonarqube is 9000
  minio:
    image: minio/minio
    entrypoint: [""]
    command: ["sh", "-c", "mkdir -p /data/mimir && minio server --address ':9900' --console-address ':9800' --quiet /data"]
    environment:
      - MINIO_ROOT_USER=mimir
      - MINIO_ROOT_PASSWORD=supersecret
    volumes:
      - minio-data:/data
    ports:
      - "9900:9900"  #SERVER
      - "9800:9800"  #CONSOLE

  mimir-1:
    image: grafana/mimir:latest
    hostname: mimir-1
    command: ["-config.file=/etc/mimir.yml"]
    volumes:
      - ${OSI_DOCKER_ROOT}\mimir\mimir.yml:/etc/mimir.yml
      - ${OSI_DOCKER_ROOT}\mimir\alertmanager-fallback-config.yml:/etc/alertmanager-fallback-config.yaml
      - mimir-1-data:/data
    depends_on:
      - minio

  mimir-2:
    image: grafana/mimir:latest
    hostname: mimir-2
    command: ["-config.file=/etc/mimir.yml"]
    volumes:
      - ${OSI_DOCKER_ROOT}\mimir\mimir.yml:/etc/mimir.yml
      - ${OSI_DOCKER_ROOT}\mimir\alertmanager-fallback-config.yml:/etc/alertmanager-fallback-config.yaml
      - mimir-2-data:/data
    depends_on:
      - minio

  mimir-3:
    image: grafana/mimir:latest
    hostname: mimir-3
    command: ["-config.file=/etc/mimir.yml"]
    volumes:
      - ${OSI_DOCKER_ROOT}\mimir\mimir.yml:/etc/mimir.yml
      - ${OSI_DOCKER_ROOT}\mimir\alertmanager-fallback-config.yml:/etc/alertmanager-fallback-config.yaml
      - mimir-3-data:/data
    depends_on:
      - minio

  load-balancer:
    image: nginx:latest
    volumes:
      - ${OSI_DOCKER_ROOT}\mimir\nginx.conf:/etc/nginx/nginx.conf:ro
    ports:
      - "9009:9009"
    depends_on:
      - mimir-1
      - mimir-2
      - mimir-3

  prometheus:
    container_name: prometheus
    image: prom/prometheus:latest
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--log.level=error'
      - '--storage.tsdb.path=/prometheus'
      - '--web.console.libraries=/usr/share/prometheus/console_libraries'
      - '--web.console.templates=/usr/share/prometheus/consoles'
    volumes:
      - ${OSI_DOCKER_ROOT}\prometheus\prometheus-mimir.yml:/etc/prometheus/prometheus.yml
      - ${OSI_DOCKER_ROOT}\mimir\rules.yml:/etc/prometheus/rules.yml
    ports:
      - "9090:9090"
    depends_on:
      - mimir-1
      - mimir-2
      - mimir-3

  otel-collector:
    container_name: opentelemetry-collector
    image: otel/opentelemetry-collector-contrib:0.91.0
    restart: always
    command:
      - "--config=/etc/otel/config.yml"
    volumes:
      - ${OSI_DOCKER_ROOT}\collector\mimir-collector.yml:/etc/otel/config.yml
    ports:
      - 1888:1888 # pprof extension
      - 8887:8887 # Prometheus metrics exposed by the Collector. "8888" is Config Server
      - 8889:8889 # Prometheus exporter metrics
      - 13133:13133 # health_check extension
      - 4317:4317 # OTLP gRPC receiver
      - 4318:4318 # OTLP http receiver
      - 55679:55679 # zpages extension
    depends_on:
      - prometheus
      - tempo
      - loki

  grafana:
    container_name: grafana
    image: grafana/grafana:latest
    environment:
      - GF_SECURITY_ADMIN=admin
      - GF_SECURITY_ADMIN_PASSWORD=j8Lhxc5RfNky
    volumes:
      - ${OSI_DOCKER_ROOT}/grafana/provisioning-dashboards.yaml:/etc/grafana/provisioning/dashboards/provisioning-dashboards.yaml:ro
      - ${OSI_DOCKER_ROOT}/grafana/provisioning-datasources.yaml:/etc/grafana/provisioning/datasources/provisioning-datasources.yaml:ro
      # Explicitly list the dashboards we want to show in the demo. We intentionally exclude dashboards that require
      # Kubernetes metrics (eg. resources or networking) and other services not available in the demo (eg. Grafana Loki).
      - ${OSI_DOCKER_ROOT}/mimir/dashboard/mimir-alertmanager.json:/var/lib/grafana/dashboards/mimir-alertmanager.json:ro
      - ${OSI_DOCKER_ROOT}/mimir/dashboard/mimir-compactor.json:/var/lib/grafana/dashboards/mimir-compactor.json:ro
      - ${OSI_DOCKER_ROOT}/mimir/dashboard/mimir-object-store.json:/var/lib/grafana/dashboards/mimir-object-store.json:ro
      - ${OSI_DOCKER_ROOT}/mimir/dashboard/mimir-overrides.json:/var/lib/grafana/dashboards/mimir-overrides.json:ro
      - ${OSI_DOCKER_ROOT}/mimir/dashboard/mimir-queries.json:/var/lib/grafana/dashboards/mimir-queries.json:ro
      - ${OSI_DOCKER_ROOT}/mimir/dashboard/mimir-reads.json:/var/lib/grafana/dashboards/mimir-reads.json:ro
      - ${OSI_DOCKER_ROOT}/mimir/dashboard/mimir-ruler.json:/var/lib/grafana/dashboards/mimir-ruler.json:ro
      - ${OSI_DOCKER_ROOT}/mimir/dashboard/mimir-tenants.json:/var/lib/grafana/dashboards/mimir-tenants.json:ro
      - ${OSI_DOCKER_ROOT}/mimir/dashboard/mimir-top-tenants.json:/var/lib/grafana/dashboards/mimir-top-tenants.json:ro
      - ${OSI_DOCKER_ROOT}/mimir/dashboard/mimir-writes.json:/var/lib/grafana/dashboards/mimir-writes.json:ro
    ports:
      - 3000:3000
    depends_on:
      - prometheus
      - tempo
      - loki
      - otel-collector

volumes:
  mimir-1-data:
  mimir-2-data:
  mimir-3-data:
  minio-data:

Prometheus:

global:
  scrape_interval: 5s
  external_labels:
    cluster: osint
    namespace: osint

rule_files:
  - /etc/prometheus/rules.yml

scrape_configs:
  - job_name: osint/mimir
    static_configs:
      - targets: ["mimir-1:8080"]
        labels:
          pod: "mimir-1"
      - targets: ["mimir-2:8080"]
        labels:
          pod: "mimir-2"
      - targets: ["mimir-3:8080"]
        labels:
          pod: "mimir-3"

remote_write:
  - url: http://load-balancer:9009/api/v1/push
    # Add X-Scope-OrgID header so that Mimir knows what tenant the remote write data should be stored in.
    # In this case, our tenant is "osint"
    # See grafana datasource
    headers:
      X-Scope-OrgID: osint

Grafana Datasource:

apiVersion: 1

datasources:
  - name: Mimir
    type: prometheus
    access: proxy
    orgId: 1
    url: http://load-balancer:9009/prometheus
    version: 1
    editable: true
    jsonData:
      httpHeaderName1: "X-Scope-OrgID"
      alertmanagerUid: "alertmanager"
    secureJsonData:
      httpHeaderValue1: "osint"
    isDefault: true
  - name: Mimir Alertmanager
    uid: alertmanager
    type: alertmanager
    access: proxy
    orgId: 1
    url: http://load-balancer:9009/
    version: 1
    editable: true
    jsonData:
      httpHeaderName1: "X-Scope-OrgID"
      implementation: "cortex"
    secureJsonData:
      httpHeaderValue1: "osint"

  - name: Tempo
    type: tempo
    access: proxy
    orgId: 1
    url: http://tempo:3200
    basicAuth: false
    isDefault: false
    version: 1
    editable: false
    apiVersion: 1
    uid: tempo
    jsonData:
      httpMethod: GET
      tracesToLogs:
        datasourceUid: 'loki'
      nodeGraph:
        enabled: true

  - name: Loki
    type: loki
    uid: loki
    access: proxy
    orgId: 1
    url: http://loki:3100
    basicAuth: false
    isDefault: false
    version: 1
    editable: false
    apiVersion: 1
    jsonData:
      derivedFields:
        -   datasourceUid: tempo
            matcherRegex: \[.+,(.+?),
            name: TraceID
            url: $${__value.raw}

Collector:

extensions:
  #https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/extension/oauth2clientauthextension/README.md
  # responsible for responding to health check calls on behalf of the collector.
  health_check:
  #  endpoint: 127.0.0.1:13133
  # fetches the collector’s performance data
  pprof:
  # serves as an http endpoint that provides live debugging data about instrumented components.
  zpages:

############ Receivers from application
receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  # batch metrics before sending to reduce API usage
  batch:
    send_batch_max_size: 1000
    send_batch_size: 100
    timeout: 10s

######### From processor, export to Logging, tracing and/or metrics servers
exporters:
  logging:
    verbosity: detailed

  #https://grafana.com/docs/mimir/latest/configure/configure-otel-collector/
  #Push metrics to prometheus mimir backend
#  prometheusremotewrite:
#    endpoint: http://localhost:9009/api/v1/push
#    external_labels:
#      env: local
  #Connection refused
  otlphttp:
    #http://localhost:9009/otlp/v1/metrics\": dial tcp 127.0.0.1:9009: connect: connection refused
    endpoint: http://localhost:9009/otlp

  # tracing to tempo
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true
  # logging to loki
  loki:
    endpoint: "http://loki:3100/loki/api/v1/push"

# The Collector pipeline.
service:
  #https://opentelemetry.io/docs/collector/configuration/#telemetry
  #Collector’s own telemetry
  telemetry:
    logs:
      level: debug
      development: true
      sampling:
        initial: 10
        thereafter: 5
      output_paths:
        - stdout
      error_output_paths:
        - stderr
    metrics:
      level: detailed

  extensions: [health_check, pprof, zpages]
  pipelines:
    # javaagent sending to otlp receiver via rpc on 4317 for all 3 metrics, traces, logs
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheusremotewrite,logging]
#      exporters: [otlphttp,logging]
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/tempo,logging]  # name here should match the exporter name for tempo which is otlp
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [loki,logging]

Spring Boot Microservice

env:
  log:
    url: http://localhost:4317 #OTLP grpc receiver
  otlp:
    url: http://localhost:4318/v1  #OTLP http receiver
    metrics-url: ${env.otlp.url}/metrics
    tracing-url: ${env.otlp.url}/traces
    logging-url: ${env.otlp.url}/logging

#https://github.com/ahmadalammar/Spring-Observability/blob/master/src/main/resources/application.yml
management:
  tracing:
    sampling:
      probability: 1.0
  endpoint:
    prometheus:
      enabled: true
    health:
      show-details: ALWAYS
  endpoints:
    web:
      exposure:
        include: "health,metrics,prometheus"
  otlp:
    metrics:
      export:
        url: ${env.otlp.metrics-url}
        step: 10s
    tracing:
      endpoint: ${env.otlp.tracing-url}
      export:
        step: 10s # Export interval set to 10 seconds
#  metrics:
#    observations:
#      key-values:
#        application: ${spring.application.name}

logging:
  level:
    io.opentelemetry: DEBUG
    io.micrometer: DEBUG
  pattern:
    level: '%5p [${spring.application.name:},%mdc{trace_id:-},%mdc{span_id:-}]'

Load Balancer:

events {
    worker_connections 1024;
}

http {
    upstream backend {
        server mimir-1:8080 max_fails=1 fail_timeout=1s;
        server mimir-2:8080 max_fails=1 fail_timeout=1s;
        server mimir-3:8080 max_fails=1 fail_timeout=1s backup;
    }

    server {
        listen 9009;
        access_log /dev/null;
        location / {
            proxy_pass http://backend;
        }
    }
}

Environment
Spring Boot 3
JDK 1.17
opentelemetry-collector-contrib:0.91.0
Docker: 4.15.0 (93002)

Additional context

2024-02-20T18:50:24.166Zinfoservice@v0.92.0/telemetry.go:86Setting up own telemetry...
2024-02-20T18:50:24.166Zinfoservice@v0.92.0/telemetry.go:159Serving metrics{"address": ":8888", "level": "Detailed"}
2024-02-20T18:50:24.166Zdebugextension@v0.92.0/extension.go:165Beta component. May change in the future.{"kind": "extension", "name": "health_check"}
2024-02-20T18:50:24.166Zdebugextension@v0.92.0/extension.go:165Beta component. May change in the future.{"kind": "extension", "name": "pprof"}
2024-02-20T18:50:24.166Zdebugextension@v0.92.0/extension.go:165Beta component. May change in the future.{"kind": "extension", "name": "zpages"}
2024-02-20T18:50:24.166Zinfoexporter@v0.92.0/exporter.go:275Deprecated component. Will be removed in future releases.{"kind": "exporter", "data_type": "metrics", "name": "logging"}
2024-02-20T18:50:24.167Zdebugexporter@v0.92.0/exporter.go:273Stable component.{"kind": "exporter", "data_type": "metrics", "name": "otlphttp"}
2024-02-20T18:50:24.167Zdebugprocessor@v0.92.0/processor.go:287Beta component. May change in the future.{"kind": "processor", "name": "batch", "pipeline": "metrics"}
2024-02-20T18:50:24.167Zdebugreceiver@v0.92.0/receiver.go:294Stable component.{"kind": "receiver", "name": "otlp", "data_type": "metrics"}
2024-02-20T18:50:24.167Zdebugexporter@v0.92.0/exporter.go:273Beta component. May change in the future.{"kind": "exporter", "data_type": "logs", "name": "loki"}
2024-02-20T18:50:24.167Zinfolokiexporter@v0.92.0/exporter.go:41using the new Loki exporter{"kind": "exporter", "data_type": "logs", "name": "loki"}
2024-02-20T18:50:24.167Zinfoexporter@v0.92.0/exporter.go:275Deprecated component. Will be removed in future releases.{"kind": "exporter", "data_type": "logs", "name": "logging"}
2024-02-20T18:50:24.168Zdebugexporter@v0.92.0/exporter.go:273Stable component.{"kind": "exporter", "data_type": "traces", "name": "otlp/tempo"}
2024-02-20T18:50:24.168Zinfoexporter@v0.92.0/exporter.go:275Deprecated component. Will be removed in future releases.{"kind": "exporter", "data_type": "traces", "name": "logging"}
2024-02-20T18:50:24.168Zdebugprocessor@v0.92.0/processor.go:287Beta component. May change in the future.{"kind": "processor", "name": "batch", "pipeline": "traces"}
2024-02-20T18:50:24.168Zdebugreceiver@v0.92.0/receiver.go:294Stable component.{"kind": "receiver", "name": "otlp", "data_type": "traces"}
2024-02-20T18:50:24.168Zdebugprocessor@v0.92.0/processor.go:287Beta component. May change in the future.{"kind": "processor", "name": "batch", "pipeline": "logs"}
2024-02-20T18:50:24.168Zdebugreceiver@v0.92.0/receiver.go:294Beta component. May change in the future.{"kind": "receiver", "name": "otlp", "data_type": "logs"}
2024-02-20T18:50:24.169Zinfoservice@v0.92.0/service.go:151Starting otelcol-contrib...{"Version": "0.92.0", "NumCPU": 16}
2024-02-20T18:50:24.169Zinfoextensions/extensions.go:34Starting extensions...
2024-02-20T18:50:24.169Zinfoextensions/extensions.go:37Extension is starting...{"kind": "extension", "name": "zpages"}
2024-02-20T18:50:24.169Zinfozpagesextension@v0.92.0/zpagesextension.go:53Registered zPages span processor on tracer provider{"kind": "extension", "name": "zpages"}
2024-02-20T18:50:24.169Zinfozpagesextension@v0.92.0/zpagesextension.go:63Registered Host's zPages{"kind": "extension", "name": "zpages"}
2024-02-20T18:50:24.170Zinfozpagesextension@v0.92.0/zpagesextension.go:75Starting zPages extension{"kind": "extension", "name": "zpages", "config": {"TCPAddr":{"Endpoint":"localhost:55679","DialerConfig":{"Timeout":0}}}}
2024-02-20T18:50:24.170Zinfoextensions/extensions.go:52Extension started.{"kind": "extension", "name": "zpages"}
2024-02-20T18:50:24.170Zinfoextensions/extensions.go:37Extension is starting...{"kind": "extension", "name": "pprof"}
2024-02-20T18:50:24.170Zinfopprofextension@v0.92.0/pprofextension.go:60Starting net/http/pprof server{"kind": "extension", "name": "pprof", "config": {"TCPAddr":{"Endpoint":"localhost:1777","DialerConfig":{"Timeout":0}},"BlockProfileFraction":0,"MutexProfileFraction":0,"SaveToFile":""}}
2024-02-20T18:50:24.170Zinfoextensions/extensions.go:52Extension started.{"kind": "extension", "name": "pprof"}
2024-02-20T18:50:24.170Zinfoextensions/extensions.go:37Extension is starting...{"kind": "extension", "name": "health_check"}
2024-02-20T18:50:24.170Zinfohealthcheckextension@v0.92.0/healthcheckextension.go:35Starting health_check extension{"kind": "extension", "name": "health_check", "config": {"Endpoint":"0.0.0.0:13133","TLSSetting":null,"CORS":null,"Auth":null,"MaxRequestBodySize":0,"IncludeMetadata":false,"ResponseHeaders":null,"Path":"/","ResponseBody":null,"CheckCollectorPipeline":{"Enabled":false,"Interval":"5m","ExporterFailureThreshold":5}}}
2024-02-20T18:50:24.170Zwarninternal@v0.92.0/warning.go:40Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks{"kind": "extension", "name": "health_check", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
go.opentelemetry.io/collector/config/internal.WarnOnUnspecifiedHost
go.opentelemetry.io/collector/config/internal@v0.92.0/warning.go:40
go.opentelemetry.io/collector/config/confighttp.(*HTTPServerSettings).ToServer
go.opentelemetry.io/collector/config/confighttp@v0.92.0/confighttp.go:323
github.com/open-telemetry/opentelemetry-collector-contrib/extension/healthcheckextension.(*healthCheckExtension).Start
github.com/open-telemetry/opentelemetry-collector-contrib/extension/healthcheckextension@v0.92.0/healthcheckextension.go:41
go.opentelemetry.io/collector/service/extensions.(*Extensions).Start
go.opentelemetry.io/collector/service@v0.92.0/extensions/extensions.go:44
go.opentelemetry.io/collector/service.(*Service).Start
go.opentelemetry.io/collector/service@v0.92.0/service.go:159
go.opentelemetry.io/collector/otelcol.(*Collector).setupConfigurationComponents
go.opentelemetry.io/collector/otelcol@v0.92.0/collector.go:190
go.opentelemetry.io/collector/otelcol.(*Collector).Run
go.opentelemetry.io/collector/otelcol@v0.92.0/collector.go:229
go.opentelemetry.io/collector/otelcol.NewCommand.func1
go.opentelemetry.io/collector/otelcol@v0.92.0/command.go:27
github.com/spf13/cobra.(*Command).execute
github.com/spf13/cobra@v1.8.0/command.go:983
github.com/spf13/cobra.(*Command).ExecuteC
github.com/spf13/cobra@v1.8.0/command.go:1115
github.com/spf13/cobra.(*Command).Execute
github.com/spf13/cobra@v1.8.0/command.go:1039
main.runInteractive
github.com/open-telemetry/opentelemetry-collector-releases/contrib/main.go:27
main.run
github.com/open-telemetry/opentelemetry-collector-releases/contrib/main_others.go:11
main.main
github.com/open-telemetry/opentelemetry-collector-releases/contrib/main.go:20
runtime.main
runtime/proc.go:267
2024-02-20T18:50:24.171Zinfoextensions/extensions.go:52Extension started.{"kind": "extension", "name": "health_check"}
2024-02-20T18:50:24.171Zinfozapgrpc/zapgrpc.go:178[core] [Channel open-telemetry/opentelemetry-collector#1] Channel created{"grpc_log": true}
2024-02-20T18:50:24.171Zinfozapgrpc/zapgrpc.go:178[core] [Channel open-telemetry/opentelemetry-collector#1] original dial target is: "tempo:4317"{"grpc_log": true}
2024-02-20T18:50:24.171Zinfozapgrpc/zapgrpc.go:178[core] [Channel open-telemetry/opentelemetry-collector#1] parsed dial target is: resolver.Target{URL:url.URL{Scheme:"tempo", Opaque:"4317", User:(*url.Userinfo)(nil), Host:"", Path:"", RawPath:"", OmitHost:false, ForceQuery:false, RawQuery:"", Fragment:"", RawFragment:""}}{"grpc_log": true}
2024-02-20T18:50:24.171Zinfozapgrpc/zapgrpc.go:178[core] [Channel open-telemetry/opentelemetry-collector#1] fallback to scheme "passthrough"{"grpc_log": true}
2024-02-20T18:50:24.171Zinfozapgrpc/zapgrpc.go:178[core] [Channel open-telemetry/opentelemetry-collector#1] parsed dial target is: passthrough:///tempo:4317{"grpc_log": true}
2024-02-20T18:50:24.171Zinfozapgrpc/zapgrpc.go:178[core] [Channel open-telemetry/opentelemetry-collector#1] Channel authority set to "tempo:4317"{"grpc_log": true}
2024-02-20T18:50:24.171Zinfozapgrpc/zapgrpc.go:178[core] [Channel open-telemetry/opentelemetry-collector#1] Resolver state updated: {
  "Addresses": [
    {
      "Addr": "tempo:4317",
      "ServerName": "",
      "Attributes": null,
      "BalancerAttributes": null,
      "Metadata": null
    }
  ],
  "Endpoints": [
    {
      "Addresses": [
        {
          "Addr": "tempo:4317",
          "ServerName": "",
          "Attributes": null,
          "BalancerAttributes": null,
          "Metadata": null
        }
      ],
      "Attributes": null
    }
  ],
  "ServiceConfig": null,
  "Attributes": null
} (resolver returned new addresses){"grpc_log": true}
2024-02-20T18:50:24.172Zinfozapgrpc/zapgrpc.go:178[core] [Channel open-telemetry/opentelemetry-collector#1] Channel switches to new LB policy "pick_first"{"grpc_log": true}
2024-02-20T18:50:24.172Zinfozapgrpc/zapgrpc.go:178[core] [pick-first-lb 0xc002df7bf0] Received new config {
  "shuffleAddressList": false
}, resolver state {
  "Addresses": [
    {
      "Addr": "tempo:4317",
      "ServerName": "",
      "Attributes": null,
      "BalancerAttributes": null,
      "Metadata": null
    }
  ],
  "Endpoints": [
    {
      "Addresses": [
        {
          "Addr": "tempo:4317",
          "ServerName": "",
          "Attributes": null,
          "BalancerAttributes": null,
          "Metadata": null
        }
      ],
      "Attributes": null
    }
  ],
  "ServiceConfig": null,
  "Attributes": null
}{"grpc_log": true}
2024-02-20T18:50:24.172Zinfozapgrpc/zapgrpc.go:178[core] [Channel open-telemetry/opentelemetry-collector#1 SubChannel open-telemetry/opentelemetry-collector#2] Subchannel created{"grpc_log": true}
2024-02-20T18:50:24.172Zinfozapgrpc/zapgrpc.go:178[core] [Channel open-telemetry/opentelemetry-collector#1] Channel Connectivity change to CONNECTING{"grpc_log": true}
2024-02-20T18:50:24.172Zinfozapgrpc/zapgrpc.go:178[core] [Channel open-telemetry/opentelemetry-collector#1] Channel exiting idle mode{"grpc_log": true}
2024-02-20T18:50:24.172Zinfozapgrpc/zapgrpc.go:178[core] [Channel open-telemetry/opentelemetry-collector#1 SubChannel open-telemetry/opentelemetry-collector#2] Subchannel Connectivity change to CONNECTING{"grpc_log": true}
2024-02-20T18:50:24.172Zinfozapgrpc/zapgrpc.go:178[core] [Channel open-telemetry/opentelemetry-collector#1 SubChannel open-telemetry/opentelemetry-collector#2] Subchannel picks a new address "tempo:4317" to connect{"grpc_log": true}
2024-02-20T18:50:24.172Zwarninternal@v0.92.0/warning.go:40Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks{"kind": "receiver", "name": "otlp", "data_type": "metrics", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
go.opentelemetry.io/collector/config/internal.WarnOnUnspecifiedHost
go.opentelemetry.io/collector/config/internal@v0.92.0/warning.go:40
go.opentelemetry.io/collector/config/configgrpc.(*GRPCServerSettings).toServerOption
go.opentelemetry.io/collector/config/configgrpc@v0.92.0/configgrpc.go:288
go.opentelemetry.io/collector/config/configgrpc.(*GRPCServerSettings).ToServer
go.opentelemetry.io/collector/config/configgrpc@v0.92.0/configgrpc.go:277
go.opentelemetry.io/collector/receiver/otlpreceiver.(*otlpReceiver).startGRPCServer
go.opentelemetry.io/collector/receiver/otlpreceiver@v0.92.0/otlp.go:86
go.opentelemetry.io/collector/receiver/otlpreceiver.(*otlpReceiver).Start
go.opentelemetry.io/collector/receiver/otlpreceiver@v0.92.0/otlp.go:172
go.opentelemetry.io/collector/internal/sharedcomponent.(*Component[...]).Start.func1
go.opentelemetry.io/collector@v0.92.0/internal/sharedcomponent/sharedcomponent.go:95
sync.(*Once).doSlow
sync/once.go:74
sync.(*Once).Do
sync/once.go:65
go.opentelemetry.io/collector/internal/sharedcomponent.(*Component[...]).Start
go.opentelemetry.io/collector@v0.92.0/internal/sharedcomponent/sharedcomponent.go:89
go.opentelemetry.io/collector/service/internal/graph.(*Graph).StartAll
go.opentelemetry.io/collector/service@v0.92.0/internal/graph/graph.go:396
go.opentelemetry.io/collector/service.(*Service).Start
go.opentelemetry.io/collector/service@v0.92.0/service.go:169
go.opentelemetry.io/collector/otelcol.(*Collector).setupConfigurationComponents
go.opentelemetry.io/collector/otelcol@v0.92.0/collector.go:190
go.opentelemetry.io/collector/otelcol.(*Collector).Run
go.opentelemetry.io/collector/otelcol@v0.92.0/collector.go:229
go.opentelemetry.io/collector/otelcol.NewCommand.func1
go.opentelemetry.io/collector/otelcol@v0.92.0/command.go:27
github.com/spf13/cobra.(*Command).execute
github.com/spf13/cobra@v1.8.0/command.go:983
github.com/spf13/cobra.(*Command).ExecuteC
github.com/spf13/cobra@v1.8.0/command.go:1115
github.com/spf13/cobra.(*Command).Execute
github.com/spf13/cobra@v1.8.0/command.go:1039
main.runInteractive
github.com/open-telemetry/opentelemetry-collector-releases/contrib/main.go:27
main.run
github.com/open-telemetry/opentelemetry-collector-releases/contrib/main_others.go:11
main.main
github.com/open-telemetry/opentelemetry-collector-releases/contrib/main.go:20
runtime.main
runtime/proc.go:267
2024-02-20T18:50:24.172Zinfootlpreceiver@v0.92.0/otlp.go:152Starting HTTP server{"kind": "receiver", "name": "otlp", "data_type": "metrics", "endpoint": "0.0.0.0:4318"}
2024-02-20T18:50:24.172Zinfozapgrpc/zapgrpc.go:178[core] [Server open-telemetry/opentelemetry-collector#3 ListenSocket open-telemetry/opentelemetry-collector#4] ListenSocket created{"grpc_log": true}
2024-02-20T18:50:24.173Zinfohealthcheck/handler.go:132Health Check state change{"kind": "extension", "name": "health_check", "status": "ready"}
2024-02-20T18:50:24.173Zinfoservice@v0.92.0/service.go:177Everything is ready. Begin running and processing data.
2024-02-20T18:50:24.173Zinfozapgrpc/zapgrpc.go:178[core] [Channel open-telemetry/opentelemetry-collector#1 SubChannel open-telemetry/opentelemetry-collector#2] Subchannel Connectivity change to READY{"grpc_log": true}
2024-02-20T18:50:24.173Zinfozapgrpc/zapgrpc.go:178[core] [pick-first-lb 0xc002df7bf0] Received SubConn state update: 0xc002df7d70, {ConnectivityState:READY ConnectionError:<nil>}{"grpc_log": true}
2024-02-20T18:50:24.173Zinfozapgrpc/zapgrpc.go:178[core] [Channel open-telemetry/opentelemetry-collector#1] Channel Connectivity change to READY{"grpc_log": true}
2024-02-20T18:51:24.173ZinfoResourceMetrics #0
Resource SchemaURL: 
Resource attributes:
     -> telemetry.sdk.name: Str(io.micrometer)
     -> telemetry.sdk.language: Str(java)
     -> telemetry.sdk.version: Str(1.12.0)
     -> service.name: Str(gdelt-ingest)
ScopeMetrics #0
ScopeMetrics SchemaURL: 
InstrumentationScope  
Metric #0
Descriptor:
     -> Name: jvm.buffer.memory.used
     -> Description: An estimate of the memory that the Java virtual machine is using for this buffer pool
     -> Unit: bytes
     -> DataType: Gauge
NumberDataPoints #0
Data point attributes:
     -> id: Str(direct)
StartTimestamp: 1970-01-01 00:00:00 +0000 UTC
Timestamp: 2024-02-20 18:51:21.859 +0000 UTC
Value: 44987.000000
Metric open-telemetry/opentelemetry-collector#1
Descriptor:
     -> Name: jvm.threads.states
     -> Description: The current number of threads
     -> Unit: threads
     -> DataType: Gauge
NumberDataPoints #0
Data point attributes:
     -> state: Str(blocked)
StartTimestamp: 1970-01-01 00:00:00 +0000 UTC
Timestamp: 2024-02-20 18:51:21.859 +0000 UTC
Value: 0.000000
Metric open-telemetry/opentelemetry-collector#2
Descriptor:
     -> Name: jvm.memory.committed
     -> Description: The amount of memory in bytes that is committed for the Java virtual machine to use
     -> Unit: bytes
     -> DataType: Gauge
NumberDataPoints #0
Data point attributes:
     -> area: Str(nonheap)
     -> id: Str(CodeCache)
StartTimestamp: 1970-01-01 00:00:00 +0000 UTC
Timestamp: 2024-02-20 18:51:21.86 +0000 UTC
Value: 15990784.000000
2024-02-20T18:51:24.177Zinfoexporterhelper/retry_sender.go:129Exporting failed. Will retry the request after interval.{"kind": "exporter", "data_type": "metrics", "name": "otlphttp", "error": "failed to make an HTTP request: Post \"http://localhost:9009/otlp/v1/metrics\": dial tcp 127.0.0.1:9009: connect: connection refused", "interval": "4.424146307s"}
2024-02-20T18:51:28.603Zdebugotlphttpexporter@v0.92.0/otlp.go:118Preparing to make HTTP request{"kind": "exporter", "data_type": "metrics", "name": "otlphttp", "url": "http://localhost:9009/otlp/v1/metrics"}
2024-02-20T18:51:28.605Zinfoexporterhelper/retry_sender.go:129Exporting failed. Will retry the request after interval.{"kind": "exporter", "data_type": "metrics", "name": "otlphttp", "error": "failed to make an HTTP request: Post \"http://localhost:9009/otlp/v1/metrics\": dial tcp 127.0.0.1:9009: connect: connection refused", "interval": "4.179256493s"}
2024-02-20T18:51:40.098Zdebugotlphttpexporter@v0.92.0/otlp.go:118Preparing to make HTTP request{"kind": "exporter", "data_type": "metrics", "name": "otlphttp", "url": "http://localhost:9009/otlp/v1/metrics"}
2024-02-20T18:51:40.099Zinfoexporterhelper/retry_sender.go:129Exporting failed. Will retry the request after interval.{"kind": "exporter", "data_type": "metrics", "name": "otlphttp", "error": "failed to make an HTTP request: Post \"http://localhost:9009/otlp/v1/metrics\": dial tcp 127.0.0.1:9009: connect: connection refused", "interval": "8.257166482s"}
2024-02-20T18:51:47.527Zdebugotlphttpexporter@v0.92.0/otlp.go:118Preparing to make HTTP request{"kind": "exporter", "data_type": "metrics", "name": "otlphttp", "url": "http://localhost:9009/otlp/v1/metrics"}
2024-02-20T18:51:47.528Zinfoexporterhelper/retry_sender.go:129Exporting failed. Will retry the request after interval.{"kind": "exporter", "data_type": "metrics", "name": "otlphttp", "error": "failed to make an HTTP request: Post \"http://localhost:9009/otlp/v1/metrics\": dial tcp 127.0.0.1:9009: connect: connection refused", "interval": "20.766239371s"}
2024-02-20T18:51:48.357Zdebugotlphttpexporter@v0.92.0/otlp.go:118Preparing to make HTTP request{"kind": "exporter", "data_type": "metrics", "name": "otlphttp", "url": "http://localhost:9009/otlp/v1/metrics"}
2024-02-20T18:51:48.359Zinfoexporterhelper/retry_sender.go:129Exporting failed. Will retry the request after interval.{"kind": "exporter", "data_type": "metrics", "name": "otlphttp", "error": "failed to make an HTTP request: Post \"http://localhost:9009/otlp/v1/metrics\": dial tcp 127.0.0.1:9009: connect: connection refused", "interval": "9.875772619s"}
2024-02-20T18:51:58.238Zdebugotlphttpexporter@v0.92.0/otlp.go:118Preparing to make HTTP request{"kind": "exporter", "data_type": "metrics", "name": "otlphttp", "url": "http://localhost:9009/otlp/v1/metrics"}
2024-02-20T18:51:58.240Zinfoexporterhelper/retry_sender.go:129Exporting failed. Will retry the request after interval.{"kind": "exporter", "data_type": "metrics", "name": "otlphttp", "error": "failed to make an HTTP request: Post \"http://localhost:9009/otlp/v1/metrics\": dial tcp 127.0.0.1:9009: connect: connection refused", "interval": "11.399770715s"}
2024-02-20T18:52:08.299Zdebugotlphttpexporter@v0.92.0/otlp.go:118Preparing to make HTTP request{"kind": "exporter", "data_type": "metrics", "name": "otlphttp", "url": "http://localhost:9009/otlp/v1/metrics"}
2024-02-20T18:52:08.300Zinfoexporterhelper/retry_sender.go:129Exporting failed. Will retry the request after interval.{"kind": "exporter", "data_type": "metrics", "name": "otlphttp", "error": "failed to make an HTTP request: Post \"http://localhost:9009/otlp/v1/metrics\": dial tcp 127.0.0.1:9009: connect: connection refused", "interval": "30.061297562s"}
2024-02-20T18:52:09.641Zdebugotlphttpexporter@v0.92.0/otlp.go:118Preparing to make HTTP request{"kind": "exporter", "data_type": "metrics", "name": "otlphttp", "url": "http://localhost:9009/otlp/v1/metrics"}
2024-02-20T18:52:09.642Zinfoexporterhelper/retry_sender.go:129Exporting failed. Will retry the request after interval.{"kind": "exporter", "data_type": "metrics", "name": "otlphttp", "error": "failed to make an HTTP request: Post \"http://localhost:9009/otlp/v1/metrics\": dial tcp 127.0.0.1:9009: connect: connection refused", "interval": "35.48513702s"}
2024-02-20T18:52:38.362Zdebugotlphttpexporter@v0.92.0/otlp.go:118Preparing to make HTTP request{"kind": "exporter", "data_type": "metrics", "name": "otlphttp", "url": "http://localhost:9009/otlp/v1/metrics"}
2024-02-20T18:52:38.363Zinfoexporterhelper/retry_sender.go:129Exporting failed. Will retry the request after interval.{"kind": "exporter", "data_type": "metrics", "name": "otlphttp", "error": "failed to make an HTTP request: Post \"http://localhost:9009/otlp/v1/metrics\": dial tcp 127.0.0.1:9009: connect: connection refused", "interval": "33.739240547s"}
2024-02-20T18:52:45.129Zdebugotlphttpexporter@v0.92.0/otlp.go:118Preparing to make HTTP request{"kind": "exporter", "data_type": "metrics", "name": "otlphttp", "url": "http://localhost:9009/otlp/v1/metrics"}
2024-02-20T18:52:45.130Zinfoexporterhelper/retry_sender.go:129Exporting failed. Will retry the request after interval.{"kind": "exporter", "data_type": "metrics", "name": "otlphttp", "error": "failed to make an HTTP request: Post \"http://localhost:9009/otlp/v1/metrics\": dial tcp 127.0.0.1:9009: connect: connection refused", "interval": "27.227569518s"}
2024-02-20T18:53:12.106Zdebugotlphttpexporter@v0.92.0/otlp.go:118Preparing to make HTTP request{"kind": "exporter", "data_type": "metrics", "name": "otlphttp", "url": "http://localhost:9009/otlp/v1/metrics"}
2024-02-20T18:53:12.107Zinfoexporterhelper/retry_sender.go:129Exporting failed. Will retry the request after interval.{"kind": "exporter", "data_type": "metrics", "name": "otlphttp", "error": "failed to make an HTTP request: Post \"http://localhost:9009/otlp/v1/metrics\": dial tcp 127.0.0.1:9009: connect: connection refused", "interval": "32.846857607s"}
2024-02-20T18:53:12.359Zdebugotlphttpexporter@v0.92.0/otlp.go:118Preparing to make HTTP request{"kind": "exporter", "data_type": "metrics", "name": "otlphttp", "url": "http://localhost:9009/otlp/v1/metrics"}
2024-02-20T18:53:12.360Zinfoexporterhelper/retry_sender.go:129Exporting failed. Will retry the request after interval.{"kind": "exporter", "data_type": "metrics", "name": "otlphttp", "error": "failed to make an HTTP request: Post \"http://localhost:9009/otlp/v1/metrics\": dial tcp 127.0.0.1:9009: connect: connection refused", "interval": "22.95444673s"}

[TylerHelmuth]

/cc @jpkrohling

[cobar79]

According to Mimir http api it is posting to the correct Distributor endpoint. Maybe it is in the Load Balancer.

[cobar79]

After reviewing prometheus config which contained a remote write with a header. I added header to the collector otlphttp exporter

  otlphttp:
    endpoint: http://localhost:9009/otlp
    headers:
      X-Scope-OrgID: osint

Still connection refused. prometheusremotewrite still provides no feed back and no results.

[jpkrohling]

@aknuds1 , @krajorama , would you be able to take a look at this one?

[cobar79]

Taking a step back, I figured it out.
Container to Container communications is via service name. Changed to load-balancer service name

  otlphttp:
    endpoint: http://load-balancer:9009/otlp
    headers:
      X-Scope-OrgID: osint