Tranform processor- expected string but got pcommon.Map

[sairamsadanala]

Component(s)

processor/transform

What happened?

Description

Fluent Bit is sending logs to otelcol-contrib otlphttp receiver and otelcol-contrib extract kubernetes.namespace_name value using Transform processor and set loki.tenant hint so that loki exporter sends logs to loki endpoint.

Steps to Reproduce

Expected Result

it should set loki.tenant =mytenant

Actual Result

    warn    ottl@v0.82.0/parser.go:211      failed to execute statement     {"kind": "processor", "name": "transform", "pipeline": "logs", "error": "expected string but got pcommon.Map",

Collector version

0.82

Environment information

Environment

OS: AWS EKS

OpenTelemetry Collector configuration

Name:         otel-hub-statefulset
Namespace:    otel-hub
Labels:       app.kubernetes.io/instance=otel-hub
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=otel-hub
              app.kubernetes.io/version=0.82.0
              helm.sh/chart=otel-hub-0.82.0_da6ca26aca3a.3
              helm.toolkit.fluxcd.io/name=otel-hub
              helm.toolkit.fluxcd.io/namespace=otel-hub
Annotations:  meta.helm.sh/release-name: otel-hub
              meta.helm.sh/release-namespace: otel-hub

Data
====
relay:
----
exporters:
  logging:
    verbosity: detailed
  loki:
    endpoint: http://XXXX/loki/api/v1/push
    retry_on_failure:
      enabled: true
      initial_interval: 1s
      max_elapsed_time: 120s
      max_interval: 300s
    sending_queue:
      storage: file_storage/psq
    tls:
      insecure: false
      insecure_skip_verify: true
  splunk_hec/logs:
    endpoint: https://splunksandbox.bpweb.bp.com:8088/services/collector
    retry_on_failure:
      enabled: true
      initial_interval: 10s
      max_elapsed_time: 60s
      max_interval: 60s
    sending_queue:
      storage: file_storage/psq
    timeout: 30s
    tls:
      insecure: false
      insecure_skip_verify: true
    token: 95c4beb5-c1b0-4735-bcd3-310cea0c9fa8
extensions:
  health_check:
    endpoint: 0.0.0.0:13133
processors:
  memory_limiter:
    check_interval: 5s
    limit_percentage: 80
    spike_limit_percentage: 25
 
  transform:
    error_mode: ignore
    log_statements:
    - context: log
      statements:
      - set(attributes["cache"], ParseJSON(body)) where IsMatch(body[""], "^\\{")
      - set(attributes["tenant"], attributes["cache"]["kubernetes"]["namespace_name"])
      - set(attributes["loki.tenant"], "tenant")
      - delete_key(attributes, "cache")
receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
        include_metadata: true
service:
  extensions:
  - health_check
  pipelines:
    logs:
      exporters:
      - logging
      - loki
      processors:
      - transform
      receivers:
      - otlp

Log output

ObservedTimestamp: 1970-01-01 00:00:00 +0000 UTC
Timestamp: 2023-09-14 15:54:35.813509554 +0000 UTC
SeverityText:
SeverityNumber: Unspecified(0)
Body: Map({"@timestamp":"2023-09-14T15:54:35.813509554+00:00","cluster":"XXX","kubernetes":{"annotations":{"XXX-linux"},"container_hash":"XX","container_image":"XX","container_name":"XX","docker_id":"XXX","host":"XX","labels":{"app":"XX","controller-revision-hash":"XX","pod-template-generation":"XX"},"namespace_name":"mynamespace","pod_id":"XX","pod_name":"XX"},"logtag":"X","message":"time=\"2XX\" level=debug msg=\"XX\" Duration=\"XX\" Method=GET RequestURL=/readiness Route=ReadinessProbe StatusCode=200 logLayer=rest_frontend requestID=XX requestSource=REST workflow=\"trident_rest=logger\"","stream":"stderr"})
Attributes:
     -> loki.tenant: Str(loki_tenant)
Trace ID:
Span ID:
Flags: 0
        {"kind": "exporter", "data_type": "logs", "name": "logging"}
2023-09-14T15:54:37.887Z        error   exporterhelper/queued_retry.go:391      Exporting failed. The error is not retryable. Dropping data.       {"kind": "exporter", "data_type": "logs", "name": "loki", "error": 
"Permanent error: HTTP 401 \"Unauthorized\": no org id", "dropped_items": 4}

Additional context

Below telemetry is being exported from Fluent Bit to Otelcol-contrib and then sending to loki endpoint.

ObservedTimestamp: 1970-01-01 00:00:00 +0000 UTC
Timestamp: 2023-09-14 15:54:35.813509554 +0000 UTC
SeverityText:
SeverityNumber: Unspecified(0)
Body: Map({"@timestamp":"2023-09-14T15:54:35.813509554+00:00","cluster":"XXX","kubernetes":{"annotations":{"XXX-linux"},"container_hash":"XX","container_image":"XX","container_name":"XX","docker_id":"XXX","host":"XX","labels":{"app":"XX","controller-revision-hash":"XX","pod-template-generation":"XX"},"namespace_name":"mynamespace","pod_id":"XX","pod_name":"XX"},"logtag":"X","message":"time="2XX" level=debug msg="XX" Duration="XX" Method=GET RequestURL=/readiness Route=ReadinessProbe StatusCode=200 logLayer=rest_frontend requestID=XX requestSource=REST workflow="trident_rest=logger"","stream":"stderr"})
Attributes:
-> loki.tenant: Str(loki_tenant)
Trace ID:
Span ID:
Flags: 0
{"kind": "exporter", "data_type": "logs", "name": "logging"}
2023-09-14T15:54:37.887Z error exporterhelper/queued_retry.go:391 Exporting failed. The error is not retryable. Dropping data. {"kind": "exporter", "data_type": "logs", "name": "loki", "error":
"Permanent error: HTTP 401 "Unauthorized": no org id", "dropped_items": 4}

1. It is working if the log body is a string. Transform processor able to extract namespace_name and set orgid.
2. It is failing the with below error if the log body is a map.
   warn ottl@v0.82.0/parser.go:211 failed to execute statement {"kind": "processor", "name": "transform", "pipeline": "logs", "error": "expected string but got pcommon.Map",

[sairamsadanala]

Expected Result
it should set loki.tenant =mynamespace

[github-actions]

Pinging code owners:

• processor/transform: @TylerHelmuth @kentquirk @bogdandrutu @evan-bradley

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[TylerHelmuth]

@sairamsadanala in the rest of the log message it should print out the exact statement that is failing, can you include that?

Also, if you want to save a statement, instead of using attributes["cache"] you can use the built in cache path:

- merge_maps(cache, ParseJSON(body), "upsert")` where IsString(body) && IsMatch(body, "^\\{")
- set(attributes["tenant"], cache["kubernetes"]["namespace_name"])
- set(attributes["loki.tenant"], "tenant")

[sairamsadanala]

26:00.268Z warn ottl@v0.82.0/parser.go:211 failed to execute statement {"kind": "processor", "name": "transform", "pipeline": "logs", "error": "expected string but got pcommon.Map", "statement": "set(attributes["cache"], ParseJSON(body)) where IsMatch(body, "^\\{")"}
26:00.268Z warn ottl@v0.82.0/parser.go:211 failed to execute statement {"kind": "processor", "name": "transform", "pipeline": "logs", "error": "expected string but got pcommon.Map", "statement": "set(attributes["cache"], ParseJSON(body)) where IsMatch(body, "^\\{")"}
26:00.268Z warn ottl@v0.82.0/parser.go:211 failed to execute statement {"kind": "processor", "name": "transform", "pipeline": "logs", "error": "expected string but got pcommon.Map", "statement": "set(attributes["cache"], ParseJSON(body)) where IsMatch(body, "^\\{")"}
26:00.268Z warn ottl@v0.82.0/parser.go:211 failed to execute statement {"kind": "processor", "name": "transform", "pipeline": "logs", "error": "expected string but got pcommon.Map", "statement": "set(attributes["cache"], ParseJSON(body)) where IsMatch(body, "^\\{")"}
26:00.268Z warn ottl@v0.82.0/parser.go:211 failed to execute statement {"kind": "processor", "name": "transform", "pipeline": "logs", "error": "expected string but got pcommon.Map", "statement": "set(attributes["cache"], ParseJSON(body)) where IsMatch(body, "^\\{")"}
26:00.268Z warn ottl@v0.82.0/parser.go:211 failed to execute statement {"kind": "processor", "name": "transform", "pipeline": "logs", "error": "expected string but got pcommon.Map", "statement": "set(attributes["cache"], ParseJSON(body)) where IsMatch(body, "^\\{")"}
26:00.268Z warn ottl@v0.82.0/parser.go:211 failed to execute statement {"kind": "processor", "name": "transform", "pipeline": "logs", "error": "expected string but got pcommon.Map", "statement": "set(attributes["cache"], ParseJSON(body)) where IsMatch(body, "^\\{")"}
26:00.268Z warn ottl@v0.82.0/parser.go:211 failed to execute statement {"kind": "processor", "name": "transform", "pipeline": "logs", "error": "expected string but got pcommon.Map", "statement": "set(attributes["cache"], ParseJSON(body)) where IsMatch(body, "^\\{")"}
26:00.269Z warn ottl@v0.82.0/parser.go:211 failed to execute statement {"kind": "processor", "name": "transform", "pipeline": "logs", "error": "expected string but got pcommon.Map", "statement": "set(attributes["cache"], ParseJSON(body)) where IsMatch(body, "^\\{")"}
26:00.269Z warn ottl@v0.82.0/parser.go:211 failed to execute statement {"kind": "processor", "name": "transform", "pipeline": "logs", "error": "expected string but got pcommon.Map", "statement": "set(attributes["cache"], ParseJSON(body)) where IsMatch(body, "^\\{")"}
26:00.269Z info LogsExporter {"kind": "exporter", "data_type": "logs", "name": "logging", "resource logs": 1, "log records": 10}
26:00.270Z info ResourceLog #0
Resource SchemaURL:
Resource attributes:
-> loki.tenant: Str(k8s.namespace.name)
-> loki.resource.labels: Str(k8s.namespace.name,k8s.pod.name,k8s.container.restart_count,k8s.pod.uid,k8s.container.name,k8s.node.name,k8s.pod.start_time,k8s.deployment.name)
ScopeLogs #0
ScopeLogs SchemaURL:
InstrumentationScope
LogRecord #0
ObservedTimestamp: 1970-01-01 00:00:00 +0000 UTC
Timestamp: 2023-09-14 18:25:56.447643346 +0000 UTC
SeverityText:
SeverityNumber: Unspecified(0)
Body: Map({"@timestamp":"25:56.447643346+00:00","cluster":"r-eu-ops-01","kubernetes":{"annotations":{"":"{"default":{"ip_addresses":["46/23"],"mac_address":"","gateway_ips":["1"],"ip_address":"46/23","gateway_ip":"1"}}","k8s.v1.cni.cncf.io/network-status":"[{\n "name": "ovn-kubernetes",\n "interface": "eth0",\n "ips": [\n "46"\n ],\n "mac": "",\n "default": true,\n "dns": {}\n}]","k8s.v1.cni.cncf.io/networks-status":"[{\n "name": "ovn-kubernetes",\n "interface": "eth0",\n "ips": [\n "46"\n ],\n "mac": "",\n "default": true,\n "dns": {}\n}]","":"1","openshift.io/deployment-config.name":"","openshift.io/deployment.name":"","openshift.io/scc":"restricted-v2","seccomp.security.alpha.kubernetes.io/pod":"runtime/default"},"container_hash":"d","container_image":"","container_name":"","docker_id":"","host":"","labels":{"app":"","app.kubernetes.io/name":"","application":"","deployment":"","deploymentconfig":"","metrics.io/exporter-type":"committime"},"namespace_name":"XXX","pod_id":"","pod_name":""},"logtag":"F","message":"DEBUG:root:Searching for images with label: app.kubernetes.io/name","stream":"stderr"})
Attributes:
-> loki.tenant: Str(tenant)
Trace ID:
Span ID:
Flags: 0
LogRecord #1
ObservedTimestamp: 1970-01-01 00:00:00 +0000 UTC
Timestamp: 2023-09-14 18:25:56.447682988 +0000 UTC
SeverityText:
SeverityNumber: Unspecified(0)
Body: Map({"@timestamp":"25:56.447682988+00:00","cluster":"r-eu-ops-01","kubernetes":{"annotations":{"":"{"default":{"ip_addresses":["46/23"],"mac_address":"","gateway_ips":["1"],"ip_address":"46/23","gateway_ip":"1"}}","k8s.v1.cni.cncf.io/network-status":"[{\n "name": "ovn-kubernetes",\n "interface": "eth0",\n "ips": [\n "46"\n ],\n "mac": "",\n "default": true,\n "dns": {}\n}]","k8s.v1.cni.cncf.io/networks-status":"[{\n "name": "ovn-kubernetes",\n "interface": "eth0",\n "ips": [\n "46"\n ],\n "mac": "",\n "default": true,\n "dns": {}\n}]","":"1","openshift.io/deployment-config.name":"","openshift.io/deployment.name":"","openshift.io/scc":"restricted-v2","seccomp.security.alpha.kubernetes.io/pod":"runtime/default"},"container_hash":"d","container_image":"","container_name":"","docker_id":"","host":"","labels":{"app":"","app.kubernetes.io/name":"","application":"","deployment":"","deploymentconfig":"","metrics.io/exporter-type":"committime"},"namespace_name":"XXX","pod_id":"","pod_name":""},"logtag":"F","message":"09-14-2023 18:25:56 DEBUG /app/committime/collector_image.py:159 generate_metrics() Searching for images with label: app.kubernetes.io/name","stream":"stderr"})
Attributes:
-> loki.tenant: Str(tenant)
Trace ID:
Span ID:
Flags: 0
LogRecord #2
ObservedTimestamp: 1970-01-01 00:00:00 +0000 UTC
Timestamp: 2023-09-14 18:25:56.530683861 +0000 UTC
SeverityText:
SeverityNumber: Unspecified(0)
Body: Map({"@timestamp":"25:56.530683862+00:00","cluster":"r-eu-ops-01","kubernetes":{"annotations":{"":"{"default":{"ip_addresses":["46/23"],"mac_address":"","gateway_ips":["1"],"ip_address":"46/23","gateway_ip":"1"}}","k8s.v1.cni.cncf.io/network-status":"[{\n "name": "ovn-kubernetes",\n "interface": "eth0",\n "ips": [\n "46"\n ],\n "mac": "",\n "default": true,\n "dns": {}\n}]","k8s.v1.cni.cncf.io/networks-status":"[{\n "name": "ovn-kubernetes",\n "interface": "eth0",\n "ips": [\n "46"\n ],\n "mac": "",\n "default": true,\n "dns": {}\n}]","":"1","openshift.io/deployment-config.name":"","openshift.io/deployment.name":"","openshift.io/scc":"restricted-v2","seccomp.security.alpha.kubernetes.io/pod":"runtime/default"},"container_hash":"d","container_image":"","container_name":"","docker_id":"","host":"","labels":{"app":"","app.kubernetes.io/name":"","application":"","deployment":"","deploymentconfig":"","metrics.io/exporter-type":"committime"},"namespace_name":"XXX","pod_id":"","pod_name":""},"logtag":"F","message":"DEBUG:root:Commit time for image provided by ''","stream":"stderr"})
Attributes:
-> loki.tenant: Str(tenant)
Trace ID:
Span ID:
Flags: 0
LogRecord #3
ObservedTimestamp: 1970-01-01 00:00:00 +0000 UTC
Timestamp: 2023-09-14 18:25:56.530734528 +0000 UTC
SeverityText:
SeverityNumber: Unspecified(0)
Body: Map({"@timestamp":"25:56.530734528+00:00","cluster":"r-eu-ops-01","kubernetes":{"annotations":{"":"{"default":{"ip_addresses":["46/23"],"mac_address":"","gateway_ips":["1"],"ip_address":"46/23","gateway_ip":"1"}}","k8s.v1.cni.cncf.io/network-status":"[{\n "name": "ovn-kubernetes",\n "interface": "eth0",\n "ips": [\n "46"\n ],\n "mac": "",\n "default": true,\n "dns": {}\n}]","k8s.v1.cni.cncf.io/networks-status":"[{\n "name": "ovn-kubernetes",\n "interface": "eth0",\n "ips": [\n "46"\n ],\n "mac": "",\n "default": true,\n "dns": {}\n}]","":"1","openshift.io/deployment-config.name":"","openshift.io/deployment.name":"","openshift.io/scc":"restricted-v2","seccomp.security.alpha.kubernetes.io/pod":"runtime/default"},"container_hash":"d","container_image":"","container_name":"","docker_id":"","host":"","labels":{"app":"","app.kubernetes.io/name":"","application":"","deployment":"","deploymentconfig":"","metrics.io/exporter-type":"committime"},"namespace_name":"XXX","pod_id":"","pod_name":""},"logtag":"F","message":"09-14-2023 18:25:56 DEBUG /app/committime/collector_image.py:142 _set_commit_time_from_annotations() Commit time for image provided by ''","stream":"stderr"})
Attributes:
-> loki.tenant: Str(tenant)
Trace ID:
Span ID:
Flags: 0
LogRecord #4
ObservedTimestamp: 1970-01-01 00:00:00 +0000 UTC
Timestamp: 2023-09-14 18:25:56.530804744 +0000 UTC
SeverityText:
SeverityNumber: Unspecified(0)
Body: Map({"@timestamp":"25:56.530804745+00:00","cluster":"r-eu-ops-01","kubernetes":{"annotations":{"":"{"default":{"ip_addresses":["46/23"],"mac_address":"","gateway_ips":["1"],"ip_address":"46/23","gateway_ip":"1"}}","k8s.v1.cni.cncf.io/network-status":"[{\n "name": "ovn-kubernetes",\n "interface": "eth0",\n "ips": [\n "46"\n ],\n "mac": "",\n "default": true,\n "dns": {}\n}]","k8s.v1.cni.cncf.io/networks-status":"[{\n "name": "ovn-kubernetes",\n "interface": "eth0",\n "ips": [\n "46"\n ],\n "mac": "",\n "default": true,\n "dns": {}\n}]","":"1","openshift.io/deployment-config.name":"","openshift.io/deployment.name":"","openshift.io/scc":"restricted-v2","seccomp.security.alpha.kubernetes.io/pod":"runtime/default"},"container_hash":"d","container_image":"","container_name":"","docker_id":"","host":"","labels":{"app":"","app.kubernetes.io/name":"","application":"","deployment":"","deploymentconfig":"","metrics.io/exporter-type":"committime"},"namespace_name":"XXX","pod_id":"","pod_name":""},"logtag":"F","message":"DEBUG:root:Commit hash for build None provided by ''","stream":"stderr"})
Attributes:
-> loki.tenant: Str(tenant)
Trace ID:
Span ID:
Flags: 0
LogRecord #5
ObservedTimestamp: 1970-01-01 00:00:00 +0000 UTC
Timestamp: 2023-09-14 18:25:56.530835555 +0000 UTC
SeverityText:
SeverityNumber: Unspecified(0)
Body: Map({"@timestamp":"25:56.530835555+00:00","cluster":"r-eu-ops-01","kubernetes":{"annotations":{"":"{"default":{"ip_addresses":["46/23"],"mac_address":"","gateway_ips":["1"],"ip_address":"46/23","gateway_ip":"1"}}","k8s.v1.cni.cncf.io/network-status":"[{\n "name": "ovn-kubernetes",\n "interface": "eth0",\n "ips": [\n "46"\n ],\n "mac": "",\n "default": true,\n "dns": {}\n}]","k8s.v1.cni.cncf.io/networks-status":"[{\n "name": "ovn-kubernetes",\n "interface": "eth0",\n "ips": [\n "46"\n ],\n "mac": "",\n "default": true,\n "dns": {}\n}]","":"1","openshift.io/deployment-config.name":"","openshift.io/deployment.name":"","openshift.io/scc":"restricted-v2","seccomp.security.alpha.kubernetes.io/pod":"runtime/default"},"container_hash":"d","container_image":"","container_name":"","docker_id":"","host":"","labels":{"app":"","app.kubernetes.io/name":"","application":"","deployment":"","deploymentconfig":"","metrics.io/exporter-type":"committime"},"namespace_name":"XXX","pod_id":"","pod_name":""},"logtag":"F","message":"09-14-2023 18:25:56 DEBUG /app/committime/collector_base.py:297 _set_commit_hash_from_annotations() Commit hash for build None provided by ''","stream":"stderr"})
Attributes:
-> loki.tenant: Str(tenant)
Trace ID:
Span ID:
Flags: 0
LogRecord #6
ObservedTimestamp: 1970-01-01 00:00:00 +0000 UTC
Timestamp: 2023-09-14 18:25:56.530899185 +0000 UTC
SeverityText:
SeverityNumber: Unspecified(0)
Body: Map({"@timestamp":"25:56.530899185+00:00","cluster":"r-eu-ops-01","kubernetes":{"annotations":{"":"{"default":{"ip_addresses":["46/23"],"mac_address":"","gateway_ips":["1"],"ip_address":"46/23","gateway_ip":"1"}}","k8s.v1.cni.cncf.io/network-status":"[{\n "name": "ovn-kubernetes",\n "interface": "eth0",\n "ips": [\n "46"\n ],\n "mac": "",\n "default": true,\n "dns": {}\n}]","k8s.v1.cni.cncf.io/networks-status":"[{\n "name": "ovn-kubernetes",\n "interface": "eth0",\n "ips": [\n "46"\n ],\n "mac": "",\n "default": true,\n "dns": {}\n}]","":"1","openshift.io/deployment-config.name":"","openshift.io/deployment.name":"","openshift.io/scc":"restricted-v2","seccomp.security.alpha.kubernetes.io/pod":"runtime/default"},"container_hash":"d","container_image":"","container_name":"","docker_id":"","host":"","labels":{"app":"","app.kubernetes.io/name":"","application":"","deployment":"","deploymentconfig":"","metrics.io/exporter-type":"committime"},"namespace_name":"XXX","pod_id":"","pod_name":""},"logtag":"F","message":"DEBUG:root:Adding metric for app ","stream":"stderr"})
Attributes:
-> loki.tenant: Str(tenant)
Trace ID:
Span ID:
Flags: 0
LogRecord #7
ObservedTimestamp: 1970-01-01 00:00:00 +0000 UTC
Timestamp: 2023-09-14 18:25:56.530925725 +0000 UTC
SeverityText:
SeverityNumber: Unspecified(0)
Body: Map({"@timestamp":"25:56.530925725+00:00","cluster":"r-eu-ops-01","kubernetes":{"annotations":{"":"{"default":{"ip_addresses":["46/23"],"mac_address":"","gateway_ips":["1"],"ip_address":"46/23","gateway_ip":"1"}}","k8s.v1.cni.cncf.io/network-status":"[{\n "name": "ovn-kubernetes",\n "interface": "eth0",\n "ips": [\n "46"\n ],\n "mac": "",\n "default": true,\n "dns": {}\n}]","k8s.v1.cni.cncf.io/networks-status":"[{\n "name": "ovn-kubernetes",\n "interface": "eth0",\n "ips": [\n "46"\n ],\n "mac": "",\n "default": true,\n "dns": {}\n}]","":"1","openshift.io/deployment-config.name":"","openshift.io/deployment.name":"","openshift.io/scc":"restricted-v2","seccomp.security.alpha.kubernetes.io/pod":"runtime/default"},"container_hash":"d","container_image":"","container_name":"","docker_id":"","host":"","labels":{"app":"","app.kubernetes.io/name":"","application":"","deployment":"","deploymentconfig":"","metrics.io/exporter-type":"committime"},"namespace_name":"XXX","pod_id":"","pod_name":""},"logtag":"F","message":"09-14-2023 18:25:56 DEBUG /app/committime/collector_image.py:199 _get_metrics_by_apps_from_images() Adding metric for app ","stream":"stderr"})
Attributes:
-> loki.tenant: Str(tenant)
Trace ID:
Span ID:
Flags: 0
LogRecord #8
ObservedTimestamp: 1970-01-01 00:00:00 +0000 UTC
Timestamp: 2023-09-14 18:25:56.530987961 +0000 UTC
SeverityText:
SeverityNumber: Unspecified(0)
Body: Map({"@timestamp":"25:56.530987961+00:00","cluster":"r-eu-ops-01","kubernetes":{"annotations":{"":"{"default":{"ip_addresses":["46/23"],"mac_address":"","gateway_ips":["1"],"ip_address":"46/23","gateway_ip":"1"}}","k8s.v1.cni.cncf.io/network-status":"[{\n "name": "ovn-kubernetes",\n "interface": "eth0",\n "ips": [\n "46"\n ],\n "mac": "",\n "default": true,\n "dns": {}\n}]","k8s.v1.cni.cncf.io/networks-status":"[{\n "name": "ovn-kubernetes",\n "interface": "eth0",\n "ips": [\n "46"\n ],\n "mac": "",\n "default": true,\n "dns": {}\n}]","":"1","openshift.io/deployment-config.name":"","openshift.io/deployment.name":"","openshift.io/scc":"restricted-v2","seccomp.security.alpha.kubernetes.io/pod":"runtime/default"},"container_hash":"d","container_image":"","container_name":"","docker_id":"","host":"","labels":{"app":"","app.kubernetes.io/name":"","application":"","deployment":"","deploymentconfig":"","metrics.io/exporter-type":"committime"},"namespace_name":"XXX","pod_id":"","pod_name":""},"logtag":"F","message":"DEBUG:root:Collected commit_timestamp{ namespace=None, app=, commit=, image_sha= } .0","stream":"stderr"})
Attributes:
-> loki.tenant: Str(tenant)
Trace ID:
Span ID:
Flags: 0
LogRecord #9
ObservedTimestamp: 1970-01-01 00:00:00 +0000 UTC
Timestamp: 2023-09-14 18:25:56.53100315 +0000 UTC
SeverityText:
SeverityNumber: Unspecified(0)
Body: Map({"@timestamp":"25:56.531003151+00:00","cluster":"r-eu-ops-01","kubernetes":{"annotations":{"":"{"default":{"ip_addresses":["46/23"],"mac_address":"","gateway_ips":["1"],"ip_address":"46/23","gateway_ip":"1"}}","k8s.v1.cni.cncf.io/network-status":"[{\n "name": "ovn-kubernetes",\n "interface": "eth0",\n "ips": [\n "46"\n ],\n "mac": "",\n "default": true,\n "dns": {}\n}]","k8s.v1.cni.cncf.io/networks-status":"[{\n "name": "ovn-kubernetes",\n "interface": "eth0",\n "ips": [\n "46"\n ],\n "mac": "",\n "default": true,\n "dns": {}\n}]","":"1","openshift.io/deployment-config.name":"","openshift.io/deployment.name":"","openshift.io/scc":"restricted-v2","seccomp.security.alpha.kubernetes.io/pod":"runtime/default"},"container_hash":"d","container_image":"","container_name":"","docker_id":"","host":"","labels":{"app":"","app.kubernetes.io/name":"","application":"","deployment":"","deploymentconfig":"","metrics.io/exporter-type":"committime"},"namespace_name":"XXX","pod_id":"","pod_name":""},"logtag":"F","message":"09-14-2023 18:25:56 DEBUG /app/committime/collector_base.py:117 collect() Collected commit_timestamp{ namespace=None, app=, commit=, image_sha= } .0","stream":"stderr"})
Attributes:
-> loki.tenant: Str(tenant)
Trace ID:
Span ID:
Flags: 0
{"kind": "exporter", "data_type": "logs", "name": "logging"}
26:00.319Z error exporterhelper/queued_retry.go:391 Exporting failed. The error is not retryable. Dropping data. {"kind": "exporter", "data_type": "logs", "name": "loki", "error": "Permanent error: HTTP 401 "Unauthorized": no org id", "dropped_items": 10}
go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/queued_retry.go:391
go.opentelemetry.io/collector/exporter/exporterhelper.(*logsExporterWithObservability).send
go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/logs.go:124
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1
go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/queued_retry.go:195
go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*persistentQueue).StartConsumers.func1
go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/internal/persistent_queue.go:55
26:01.194Z warn ottl@v0.82.0/parser.go:211 failed to execute statement {"kind": "processor", "name": "transform", "pipeline": "logs", "error": "expected string but got pcommon.Map", "statement": "set(attributes["cache"], ParseJSON(body)) where IsMatch(body, "^\\{")"}
26:01.195Z warn ottl@v0.82.0/parser.go:211 failed to execute statement {"kind": "processor", "name": "transform", "pipeline": "logs", "error": "expected string but got pcommon.Map", "statement": "set(attributes["cache"], ParseJSON(body)) where IsMatch(body, "^\\{")"}
26:01.196Z warn ottl@v0.82.0/parser.go:211 failed to execute statement {"kind": "processor", "name": "transform", "pipeline": "logs", "error": "expected string but got pcommon.Map", "statement": "set(attributes["cache"], ParseJSON(body)) where IsMatch(body, "^\\{")"}
26:01.196Z warn ottl@v0.82.0/parser.go:211 failed to execute statement {"kind": "processor", "name": "transform", "pipeline": "logs", "error": "expected string but got pcommon.Map", "statement": "set(attributes["cache"], ParseJSON(body)) where IsMatch(body, "^\\{")"}
26:01.196Z info LogsExporter {"kind": "exporter", "data_type": "logs", "name": "logging", "resource logs": 1, "log records": 4}

[TylerHelmuth]

Something is wrong with this section of your condition: IsMatch(body[""], "^\\{"). IsMatch takes a string and is complaining about body[""] resolving to a map. It looks like body is already a map, so you don't need to use ParseJSON

[sairamsadanala]

Thanks @TylerHelmuth. I was trying different functions and might have given you wrong one.

body.string function converted map to string which solved my problem.
transform:
error_mode: ignore
log_statements:
- context: log
statements:
- set(attributes["cache"], ParseJSON(body.string)) where IsMatch(body.string, "^\{")
- set(attributes["tenant"], attributes["cache"]["kubernetes"]["namespace_name"])
- set(attributes["loki.tenant"], "tenant")
- delete_key(attributes, "cache")