prometheusremotewrite exporter doesn't work well with groupbyattrs processor?

[newly12]

Describe the bug
When groupbyattrs processor is enabled and use prometheusremotewrite exporter in metrics pipeine, there will be failures/errors in prometheus translator at here probably because flags likepmetric.MetricAggregationTemporalityCumulative are removed or ignored by groupbyattrs processor.

Steps to reproduce
Please use the given config below to reproduce this error Permanent error: invalid temporality and type combination

What did you expect to see?
things to work while having these 2 components in a metrics pipeline.

What did you see instead?

2022-05-12T22:46:09.098+0800    error   exporterhelper/queued_retry.go:183      Exporting failed. The error is not retryable. Dropping data.    {"kind": "exporter", "name": "prometheusremotewrite", "error": "Permanent error: invalid temporality and type combination; invalid temporality and type combination; invalid temporality and type combination; invalid temporality and type combination; Permanent error: Permanent error: remote write returned HTTP status 404 Not Found; err = <nil>: 404 page not found\n", "errorCauses": [{"error": "Permanent error: invalid temporality and type combination; invalid temporality and type combination; invalid temporality and type combination; invalid temporality and type combination"}, {"error": "Permanent error: Permanent error: remote write returned HTTP status 404 Not Found; err = <nil>: 404 page not found\n"}], "dropped_items": 15}

What version did you use?
Version: v0.50.0

What config did you use?
Config: (e.g. the yaml config file)

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: kubernetes-cadvisor-metrics
          static_configs:
            - targets:
                - localhost:8888
              labels:
                k: v

exporters:
  prometheusremotewrite:
    endpoint: http://localhost:8888


processors:
  groupbyattrs:
    keys:
      - k

service:
  telemetry:
    logs:
      level: debug
    metrics:
      level: detailed
      address: 127.0.0.1:8888
  pipelines:
    metrics:
      receivers: [prometheus]
      processors: [groupbyattrs]
      exporters: [ prometheusremotewrite]

Environment
OS: MacOS 12.3.1
Compiler(if manually compiled): go 1.17.8

Additional context
Add any other context about the problem here.

[bogdandrutu]

@newly12 indeed this looks like a bug, but I am curious why do you need this processor in your pipeline? This does exactly opposite of what prometheus remote write protocol needs which is to "flatten" all the attributes as labels for each data point.

[dmitryax]

This should be fixed by #9088

[newly12]

Thanks for the quick response.

We are leveraging prometheus receiver to pull cadvisor metrics and also enrich the metrics given the namespace and pod(from metrics label), currently k8sattrs processor only supports resource attributes, not attributes from a metric, the easiest way I can think of is using gropubyattrs processor to group namespace and pod so they appear in the resource attributes. I appreciated if any suggestion.

[newly12]

Thanks for the quick response.

We are leveraging prometheus receiver to pull cadvisor metrics and also enrich the metrics given the namespace and pod(from metrics label), currently k8sattrs processor only supports resource attributes, not attributes from a metric, the easiest way I can think of is using gropubyattrs processor to group namespace and pod so they appear in the resource attributes. I appreciated if any suggestion.

For who may be interested, other issues with this approach #10021 #10023