[target allocator] Generate secret containing require TLS creds for scraping.

[tcolgate]

if service monitors include the .spec.tlsConfig scraping fails unless the Collector has suitable volumes and volume mounts to bring in the correct credentials. The Prometheus Operator does generate such a secret (which can be reused), but ideally the target allocator would provide it's own.

[matej-g]

Related to #1669

[rlanore]

am i hit this if i see this log into collecor:

HTTP client: unable to load specified CA cert /etc/prometheus/certs/secret_monitoring_prometheus-kube-prometheus-admission_ca: open /etc/prometheus/certs/secret_monitoring_prometheus-kube-prometheus-kube-prometheus-admission_ca: no such file or directory"

From curl TA at /scrap_confis i have this configuration for prometheus operator

  "serviceMonitor/monitoring/prometheus-kube-prometheus-operator/0": {
    "enable_http2": true,
    "follow_redirects": true,
    "honor_labels": true,
    "honor_timestamps": true,
    "job_name": "serviceMonitor/monitoring/prometheus-kube-prometheus-operator/0",
... ...
    "scheme": "https",
    "scrape_interval": "30s",
    "scrape_timeout": "10s",
    "tls_config": {
      "ca_file": "/etc/prometheus/certs/secret_monitoring_prometheus-kube-prometheus-admission_ca",
      "insecure_skip_verify": false,
      "server_name": "prometheus-kube-prometheus-operator"
    }
  },

[jaronoff97]

@matej-g would you be able to work on this if you get a chance? if not i'll poll the group and see if anyone else can!

[Jojoooo1]

also having the same problem, trying to migrate from kube prometheus stack

[matej-g]

Hey, @jaronoff97 let me take a look at this

[jaronoff97]

@matej-g heads up as we're changing some of the logic for this here

[jouve]

in #1710, work was done generate the store struct and dump it in the generated config.

For this issue, the same struct must be reconciled in a k8s Secret (similar to https://github.com/prometheus-operator/prometheus-operator/blob/main/pkg/prometheus/server/operator.go#L1000) and this Secret mounted in the collector pods

[jouve]

An alternative would be to put the content of the certs in the generated config without the indirection to a file (ca instead of ca_file, cert instead of cert_file, etc)

[rtrevi]

An alternative would be to put the content of the certs in the generated config without the indirection to a file (ca instead of ca_file, cert instead of cert_file, etc)

This would support PodMonitors as well since those intentionally disable file references.

And the suggested workaround of using ScrapeClasses is not available in this operator.