Creation of Package Policies through API dont work for complex policy sample

[gizas]

Kibana version:
8.3.2
Elasticsearch version:
8.3.2

Server OS version:

Browser version:
Chrome 104.0.5112.79

Browser OS version:
Version 104.0.5112.79 (Official Build) (arm64)

Original install method (e.g. download page, yum, from source, etc.):
Elastic Cloud
Describe the bug:
I am using the Fleet/Documentation API commands of given doc https://docs.google.com/document/d/1HP_UJQ8ntQijzagH4R-B8hSP3MDPwpVEHv0s54ysu8Q

Goal is to install the kubernetes Integration package inside a given Agent.

For the moment I have managed to authenticate to Elastic Cloud with following api config

POST /_security/api_key
{
  "name": "test", 
  "role_descriptors": {
    "test": {
    "cluster": [],
    "indices": [],
    "applications": [
      {
        "application": "kibana-.kibana",
        "privileges": [
          "feature_fleet.all",
          "feature_fleetv2.all"
        ],
        "resources": [
          "*"
        ]
      }
    ]}
  }
}

My commands are successfull for installing nginx and also the package_policy given in the example.

Now I am trying to install following complex package policy:
k8spolicy.json.txt

Command:

 curl --location --request POST 'https://gizas-test.kb.us-central1.gcp.cloud.es.io:9243/api/fleet/package_policies' --header 'Authorization: ApiKey <mykey>' --header 'kbn-xsrf: true' --header 'Content-Type: application/json' \     
--data-raw '<all contents from file>'

But no error is produced and even nothing when I try -vvv verbose output.
Can you please help me troubleshoot and create package policy for k8s integration?

Agent with "policy_id": "d42d3030-1a3a-11ed-bda0-8feb86b4d5f6" already exists in my cluster

Steps to reproduce:

1. Just try to create the above package policy

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[kpollich]

Related to #132263

[nchaulet]

@gizas the following request worked on my side I removed the version from the payload

curl --request POST \
  --url http://localhost:5601/api/fleet/package_policies \
  --header 'Authorization: ApiKey VmNmR3E0SUJRVnV1WFQ0bC1fMlU6QUVjbmp2Ui1TakNZb2dRalhramJhQQ==' \
  --header 'Content-Type: application/json' \
  --header 'kbn-xsrf: xx' \
  --data '{
  "id": "48db9a8e-488b-4c2a-b3b0-4dde30bdc290",
  "name": "kubernetes-test2",
  "namespace": "default",
  "description": "",
  "package": {
    "name": "kubernetes",
    "title": "Kubernetes",
    "version": "1.21.2"
  },
  "policy_id": "342352e0-1d9b-11ed-ae20-dbcc242c1605",
  "output_id": "",
  "inputs": [
    {
      "type": "kubernetes/metrics",
      "policy_template": "kubelet",
      "enabled": true,
      "streams": [
        {
          "enabled": true,
          "data_stream": {
            "type": "metrics",
            "dataset": "kubernetes.container"
          },
          "vars": {
            "add_metadata": {
              "value": true,
              "type": "bool"
            },
            "bearer_token_file": {
              "value": "/var/run/secrets/kubernetes.io/serviceaccount/token",
              "type": "text"
            },
            "hosts": {
              "value": [
                "https://${env.NODE_NAME}:10250"
              ],
              "type": "text"
            },
            "period": {
              "value": "10s",
              "type": "text"
            },
            "ssl.verification_mode": {
              "value": "none",
              "type": "text"
            },
            "add_resource_metadata_config": {
              "value": "# add_resource_metadata:\n#   namespace:\n#     include_labels: [\"namespacelabel1\"]\n#   node:\n#     include_labels: [\"nodelabel2\"]\n#     include_annotations: [\"nodeannotation1\"]\n#   deployment: false\n",
              "type": "yaml"
            },
            "ssl.certificate_authorities": {
              "value": [],
              "type": "text"
            }
          },
          "id": "kubernetes/metrics-kubernetes.container-48db9a8e-488b-4c2a-b3b0-4dde30bdc268",
          "compiled_stream": {
            "metricsets": [
              "container"
            ],
            "add_metadata": true,
            "hosts": [
              "https://${env.NODE_NAME}:10250"
            ],
            "period": "10s",
            "bearer_token_file": "/var/run/secrets/kubernetes.io/serviceaccount/token",
            "ssl.verification_mode": "none"
          }
        },
        {
          "enabled": true,
          "data_stream": {
            "type": "metrics",
            "dataset": "kubernetes.node"
          },
          "vars": {
            "add_metadata": {
              "value": true,
              "type": "bool"
            },
            "bearer_token_file": {
              "value": "/var/run/secrets/kubernetes.io/serviceaccount/token",
              "type": "text"
            },
            "hosts": {
              "value": [
                "https://${env.NODE_NAME}:10250"
              ],
              "type": "text"
            },
            "period": {
              "value": "10s",
              "type": "text"
            },
            "ssl.verification_mode": {
              "value": "none",
              "type": "text"
            },
            "ssl.certificate_authorities": {
              "value": [],
              "type": "text"
            }
          },
          "id": "kubernetes/metrics-kubernetes.node-48db9a8e-488b-4c2a-b3b0-4dde30bdc268",
          "compiled_stream": {
            "metricsets": [
              "node"
            ],
            "add_metadata": true,
            "hosts": [
              "https://${env.NODE_NAME}:10250"
            ],
            "period": "10s",
            "bearer_token_file": "/var/run/secrets/kubernetes.io/serviceaccount/token",
            "ssl.verification_mode": "none"
          }
        },
        {
          "enabled": true,
          "data_stream": {
            "type": "metrics",
            "dataset": "kubernetes.pod"
          },
          "vars": {
            "add_metadata": {
              "value": true,
              "type": "bool"
            },
            "bearer_token_file": {
              "value": "/var/run/secrets/kubernetes.io/serviceaccount/token",
              "type": "text"
            },
            "hosts": {
              "value": [
                "https://${env.NODE_NAME}:10250"
              ],
              "type": "text"
            },
            "period": {
              "value": "10s",
              "type": "text"
            },
            "ssl.verification_mode": {
              "value": "none",
              "type": "text"
            },
            "ssl.certificate_authorities": {
              "value": [],
              "type": "text"
            },
            "add_resource_metadata_config": {
              "value": "# add_resource_metadata:\n#   namespace:\n#     include_labels: [\"namespacelabel1\"]\n#   node:\n#     include_labels: [\"nodelabel2\"]\n#     include_annotations: [\"nodeannotation1\"]\n#   deployment: false\n",
              "type": "yaml"
            }
          },
          "id": "kubernetes/metrics-kubernetes.pod-48db9a8e-488b-4c2a-b3b0-4dde30bdc268",
          "compiled_stream": {
            "metricsets": [
              "pod"
            ],
            "add_metadata": true,
            "hosts": [
              "https://${env.NODE_NAME}:10250"
            ],
            "period": "10s",
            "bearer_token_file": "/var/run/secrets/kubernetes.io/serviceaccount/token",
            "ssl.verification_mode": "none"
          }
        },
        {
          "enabled": true,
          "data_stream": {
            "type": "metrics",
            "dataset": "kubernetes.system"
          },
          "vars": {
            "add_metadata": {
              "value": true,
              "type": "bool"
            },
            "bearer_token_file": {
              "value": "/var/run/secrets/kubernetes.io/serviceaccount/token",
              "type": "text"
            },
            "hosts": {
              "value": [
                "https://${env.NODE_NAME}:10250"
              ],
              "type": "text"
            },
            "period": {
              "value": "10s",
              "type": "text"
            },
            "ssl.verification_mode": {
              "value": "none",
              "type": "text"
            },
            "ssl.certificate_authorities": {
              "value": [],
              "type": "text"
            }
          },
          "id": "kubernetes/metrics-kubernetes.system-48db9a8e-488b-4c2a-b3b0-4dde30bdc268",
          "compiled_stream": {
            "metricsets": [
              "system"
            ],
            "add_metadata": true,
            "hosts": [
              "https://${env.NODE_NAME}:10250"
            ],
            "period": "10s",
            "bearer_token_file": "/var/run/secrets/kubernetes.io/serviceaccount/token",
            "ssl.verification_mode": "none"
          }
        },
        {
          "enabled": true,
          "data_stream": {
            "type": "metrics",
            "dataset": "kubernetes.volume"
          },
          "vars": {
            "add_metadata": {
              "value": true,
              "type": "bool"
            },
            "bearer_token_file": {
              "value": "/var/run/secrets/kubernetes.io/serviceaccount/token",
              "type": "text"
            },
            "hosts": {
              "value": [
                "https://${env.NODE_NAME}:10250"
              ],
              "type": "text"
            },
            "period": {
              "value": "10s",
              "type": "text"
            },
            "ssl.verification_mode": {
              "value": "none",
              "type": "text"
            },
            "ssl.certificate_authorities": {
              "value": [],
              "type": "text"
            }
          },
          "id": "kubernetes/metrics-kubernetes.volume-48db9a8e-488b-4c2a-b3b0-4dde30bdc268",
          "compiled_stream": {
            "metricsets": [
              "volume"
            ],
            "add_metadata": true,
            "hosts": [
              "https://${env.NODE_NAME}:10250"
            ],
            "period": "10s",
            "bearer_token_file": "/var/run/secrets/kubernetes.io/serviceaccount/token",
            "ssl.verification_mode": "none"
          }
        }
      ]
    },
    {
      "type": "kubernetes/metrics",
      "policy_template": "kube-state-metrics",
      "enabled": true,
      "streams": [
        {
          "enabled": true,
          "data_stream": {
            "type": "metrics",
            "dataset": "kubernetes.state_container"
          },
          "vars": {
            "add_metadata": {
              "value": true,
              "type": "bool"
            },
            "hosts": {
              "value": [
                "kube-state-metrics:8080"
              ],
              "type": "text"
            },
            "leaderelection": {
              "value": true,
              "type": "bool"
            },
            "period": {
              "value": "10s",
              "type": "text"
            },
            "node": {
              "type": "text"
            },
            "namespace": {
              "type": "text"
            },
            "bearer_token_file": {
              "value": "/var/run/secrets/kubernetes.io/serviceaccount/token",
              "type": "text"
            },
            "ssl.certificate_authorities": {
              "value": [],
              "type": "text"
            },
            "add_resource_metadata_config": {
              "value": "# add_resource_metadata:\n#   namespace:\n#     include_labels: [\"namespacelabel1\"]\n#   node:\n#     include_labels: [\"nodelabel2\"]\n#     include_annotations: [\"nodeannotation1\"]\n#   deployment: false\n",
              "type": "yaml"
            }
          },
          "id": "kubernetes/metrics-kubernetes.state_container-48db9a8e-488b-4c2a-b3b0-4dde30bdc268",
          "compiled_stream": {
            "metricsets": [
              "state_container"
            ],
            "add_metadata": true,
            "hosts": [
              "kube-state-metrics:8080"
            ],
            "period": "10s",
            "condition": "${kubernetes_leaderelection.leader} == true",
            "bearer_token_file": "/var/run/secrets/kubernetes.io/serviceaccount/token"
          }
        },
        {
          "enabled": true,
          "data_stream": {
            "type": "metrics",
            "dataset": "kubernetes.state_cronjob"
          },
          "vars": {
            "add_metadata": {
              "value": true,
              "type": "bool"
            },
            "hosts": {
              "value": [
                "kube-state-metrics:8080"
              ],
              "type": "text"
            },
            "leaderelection": {
              "value": true,
              "type": "bool"
            },
            "period": {
              "value": "10s",
              "type": "text"
            },
            "node": {
              "type": "text"
            },
            "namespace": {
              "type": "text"
            },
            "bearer_token_file": {
              "value": "/var/run/secrets/kubernetes.io/serviceaccount/token",
              "type": "text"
            },
            "ssl.certificate_authorities": {
              "value": [],
              "type": "text"
            }
          },
          "id": "kubernetes/metrics-kubernetes.state_cronjob-48db9a8e-488b-4c2a-b3b0-4dde30bdc268",
          "compiled_stream": {
            "metricsets": [
              "state_cronjob"
            ],
            "add_metadata": true,
            "hosts": [
              "kube-state-metrics:8080"
            ],
            "period": "10s",
            "condition": "${kubernetes_leaderelection.leader} == true",
            "bearer_token_file": "/var/run/secrets/kubernetes.io/serviceaccount/token"
          }
        },
        {
          "enabled": true,
          "data_stream": {
            "type": "metrics",
            "dataset": "kubernetes.state_daemonset"
          },
          "vars": {
            "add_metadata": {
              "value": true,
              "type": "bool"
            },
            "hosts": {
              "value": [
                "kube-state-metrics:8080"
              ],
              "type": "text"
            },
            "leaderelection": {
              "value": true,
              "type": "bool"
            },
            "period": {
              "value": "10s",
              "type": "text"
            },
            "node": {
              "type": "text"
            },
            "namespace": {
              "type": "text"
            },
            "bearer_token_file": {
              "value": "/var/run/secrets/kubernetes.io/serviceaccount/token",
              "type": "text"
            },
            "ssl.certificate_authorities": {
              "value": [],
              "type": "text"
            }
          },
          "id": "kubernetes/metrics-kubernetes.state_daemonset-48db9a8e-488b-4c2a-b3b0-4dde30bdc268",
          "compiled_stream": {
            "metricsets": [
              "state_daemonset"
            ],
            "add_metadata": true,
            "hosts": [
              "kube-state-metrics:8080"
            ],
            "period": "10s",
            "condition": "${kubernetes_leaderelection.leader} == true",
            "bearer_token_file": "/var/run/secrets/kubernetes.io/serviceaccount/token"
          }
        },
        {
          "enabled": true,
          "data_stream": {
            "type": "metrics",
            "dataset": "kubernetes.state_deployment"
          },
          "vars": {
            "add_metadata": {
              "value": true,
              "type": "bool"
            },
            "hosts": {
              "value": [
                "kube-state-metrics:8080"
              ],
              "type": "text"
            },
            "leaderelection": {
              "value": true,
              "type": "bool"
            },
            "period": {
              "value": "10s",
              "type": "text"
            },
            "node": {
              "type": "text"
            },
            "namespace": {
              "type": "text"
            },
            "bearer_token_file": {
              "value": "/var/run/secrets/kubernetes.io/serviceaccount/token",
              "type": "text"
            },
            "ssl.certificate_authorities": {
              "value": [],
              "type": "text"
            }
          },
          "id": "kubernetes/metrics-kubernetes.state_deployment-48db9a8e-488b-4c2a-b3b0-4dde30bdc268",
          "compiled_stream": {
            "metricsets": [
              "state_deployment"
            ],
            "add_metadata": true,
            "hosts": [
              "kube-state-metrics:8080"
            ],
            "period": "10s",
            "condition": "${kubernetes_leaderelection.leader} == true",
            "bearer_token_file": "/var/run/secrets/kubernetes.io/serviceaccount/token"
          }
        },
        {
          "enabled": true,
          "data_stream": {
            "type": "metrics",
            "dataset": "kubernetes.state_job"
          },
          "vars": {
            "add_metadata": {
              "value": true,
              "type": "bool"
            },
            "hosts": {
              "value": [
                "kube-state-metrics:8080"
              ],
              "type": "text"
            },
            "leaderelection": {
              "value": true,
              "type": "bool"
            },
            "period": {
              "value": "10s",
              "type": "text"
            },
            "node": {
              "type": "text"
            },
            "namespace": {
              "type": "text"
            },
            "bearer_token_file": {
              "value": "/var/run/secrets/kubernetes.io/serviceaccount/token",
              "type": "text"
            },
            "ssl.certificate_authorities": {
              "value": [],
              "type": "text"
            }
          },
          "id": "kubernetes/metrics-kubernetes.state_job-48db9a8e-488b-4c2a-b3b0-4dde30bdc268",
          "compiled_stream": {
            "metricsets": [
              "state_job"
            ],
            "add_metadata": true,
            "hosts": [
              "kube-state-metrics:8080"
            ],
            "period": "10s",
            "condition": "${kubernetes_leaderelection.leader} == true",
            "bearer_token_file": "/var/run/secrets/kubernetes.io/serviceaccount/token"
          }
        },
        {
          "enabled": true,
          "data_stream": {
            "type": "metrics",
            "dataset": "kubernetes.state_node"
          },
          "vars": {
            "add_metadata": {
              "value": true,
              "type": "bool"
            },
            "hosts": {
              "value": [
                "kube-state-metrics:8080"
              ],
              "type": "text"
            },
            "leaderelection": {
              "value": true,
              "type": "bool"
            },
            "period": {
              "value": "10s",
              "type": "text"
            },
            "node": {
              "type": "text"
            },
            "namespace": {
              "type": "text"
            },
            "bearer_token_file": {
              "value": "/var/run/secrets/kubernetes.io/serviceaccount/token",
              "type": "text"
            },
            "ssl.certificate_authorities": {
              "value": [],
              "type": "text"
            }
          },
          "id": "kubernetes/metrics-kubernetes.state_node-48db9a8e-488b-4c2a-b3b0-4dde30bdc268",
          "compiled_stream": {
            "metricsets": [
              "state_node"
            ],
            "add_metadata": true,
            "hosts": [
              "kube-state-metrics:8080"
            ],
            "period": "10s",
            "condition": "${kubernetes_leaderelection.leader} == true",
            "bearer_token_file": "/var/run/secrets/kubernetes.io/serviceaccount/token"
          }
        },
        {
          "enabled": true,
          "data_stream": {
            "type": "metrics",
            "dataset": "kubernetes.state_persistentvolume"
          },
          "vars": {
            "add_metadata": {
              "value": true,
              "type": "bool"
            },
            "hosts": {
              "value": [
                "kube-state-metrics:8080"
              ],
              "type": "text"
            },
            "leaderelection": {
              "value": true,
              "type": "bool"
            },
            "period": {
              "value": "10s",
              "type": "text"
            },
            "node": {
              "type": "text"
            },
            "namespace": {
              "type": "text"
            },
            "bearer_token_file": {
              "value": "/var/run/secrets/kubernetes.io/serviceaccount/token",
              "type": "text"
            },
            "ssl.certificate_authorities": {
              "value": [],
              "type": "text"
            }
          },
          "id": "kubernetes/metrics-kubernetes.state_persistentvolume-48db9a8e-488b-4c2a-b3b0-4dde30bdc268",
          "compiled_stream": {
            "metricsets": [
              "state_persistentvolume"
            ],
            "add_metadata": true,
            "hosts": [
              "kube-state-metrics:8080"
            ],
            "period": "10s",
            "condition": "${kubernetes_leaderelection.leader} == true",
            "bearer_token_file": "/var/run/secrets/kubernetes.io/serviceaccount/token"
          }
        },
        {
          "enabled": true,
          "data_stream": {
            "type": "metrics",
            "dataset": "kubernetes.state_persistentvolumeclaim"
          },
          "vars": {
            "add_metadata": {
              "value": true,
              "type": "bool"
            },
            "hosts": {
              "value": [
                "kube-state-metrics:8080"
              ],
              "type": "text"
            },
            "leaderelection": {
              "value": true,
              "type": "bool"
            },
            "period": {
              "value": "10s",
              "type": "text"
            },
            "node": {
              "type": "text"
            },
            "namespace": {
              "type": "text"
            },
            "bearer_token_file": {
              "value": "/var/run/secrets/kubernetes.io/serviceaccount/token",
              "type": "text"
            },
            "ssl.certificate_authorities": {
              "value": [],
              "type": "text"
            }
          },
          "id": "kubernetes/metrics-kubernetes.state_persistentvolumeclaim-48db9a8e-488b-4c2a-b3b0-4dde30bdc268",
          "compiled_stream": {
            "metricsets": [
              "state_persistentvolumeclaim"
            ],
            "add_metadata": true,
            "hosts": [
              "kube-state-metrics:8080"
            ],
            "period": "10s",
            "condition": "${kubernetes_leaderelection.leader} == true",
            "bearer_token_file": "/var/run/secrets/kubernetes.io/serviceaccount/token"
          }
        },
        {
          "enabled": true,
          "data_stream": {
            "type": "metrics",
            "dataset": "kubernetes.state_pod"
          },
          "vars": {
            "add_metadata": {
              "value": true,
              "type": "bool"
            },
            "hosts": {
              "value": [
                "kube-state-metrics:8080"
              ],
              "type": "text"
            },
            "leaderelection": {
              "value": true,
              "type": "bool"
            },
            "period": {
              "value": "10s",
              "type": "text"
            },
            "node": {
              "type": "text"
            },
            "namespace": {
              "type": "text"
            },
            "bearer_token_file": {
              "value": "/var/run/secrets/kubernetes.io/serviceaccount/token",
              "type": "text"
            },
            "ssl.certificate_authorities": {
              "value": [],
              "type": "text"
            },
            "add_resource_metadata_config": {
              "value": "# add_resource_metadata:\n#   namespace:\n#     include_labels: [\"namespacelabel1\"]\n#   node:\n#     include_labels: [\"nodelabel2\"]\n#     include_annotations: [\"nodeannotation1\"]\n#   deployment: false\n",
              "type": "yaml"
            }
          },
          "id": "kubernetes/metrics-kubernetes.state_pod-48db9a8e-488b-4c2a-b3b0-4dde30bdc268",
          "compiled_stream": {
            "metricsets": [
              "state_pod"
            ],
            "add_metadata": true,
            "hosts": [
              "kube-state-metrics:8080"
            ],
            "period": "10s",
            "condition": "${kubernetes_leaderelection.leader} == true",
            "bearer_token_file": "/var/run/secrets/kubernetes.io/serviceaccount/token"
          }
        },
        {
          "enabled": true,
          "data_stream": {
            "type": "metrics",
            "dataset": "kubernetes.state_replicaset"
          },
          "vars": {
            "add_metadata": {
              "value": true,
              "type": "bool"
            },
            "hosts": {
              "value": [
                "kube-state-metrics:8080"
              ],
              "type": "text"
            },
            "leaderelection": {
              "value": true,
              "type": "bool"
            },
            "period": {
              "value": "10s",
              "type": "text"
            },
            "node": {
              "type": "text"
            },
            "namespace": {
              "type": "text"
            },
            "bearer_token_file": {
              "value": "/var/run/secrets/kubernetes.io/serviceaccount/token",
              "type": "text"
            },
            "ssl.certificate_authorities": {
              "value": [],
              "type": "text"
            }
          },
          "id": "kubernetes/metrics-kubernetes.state_replicaset-48db9a8e-488b-4c2a-b3b0-4dde30bdc268",
          "compiled_stream": {
            "metricsets": [
              "state_replicaset"
            ],
            "add_metadata": true,
            "hosts": [
              "kube-state-metrics:8080"
            ],
            "period": "10s",
            "condition": "${kubernetes_leaderelection.leader} == true",
            "bearer_token_file": "/var/run/secrets/kubernetes.io/serviceaccount/token"
          }
        },
        {
          "enabled": true,
          "data_stream": {
            "type": "metrics",
            "dataset": "kubernetes.state_resourcequota"
          },
          "vars": {
            "add_metadata": {
              "value": true,
              "type": "bool"
            },
            "hosts": {
              "value": [
                "kube-state-metrics:8080"
              ],
              "type": "text"
            },
            "leaderelection": {
              "value": true,
              "type": "bool"
            },
            "period": {
              "value": "10s",
              "type": "text"
            },
            "node": {
              "type": "text"
            },
            "namespace": {
              "type": "text"
            },
            "bearer_token_file": {
              "value": "/var/run/secrets/kubernetes.io/serviceaccount/token",
              "type": "text"
            },
            "ssl.certificate_authorities": {
              "value": [],
              "type": "text"
            }
          },
          "id": "kubernetes/metrics-kubernetes.state_resourcequota-48db9a8e-488b-4c2a-b3b0-4dde30bdc268",
          "compiled_stream": {
            "metricsets": [
              "state_resourcequota"
            ],
            "add_metadata": true,
            "hosts": [
              "kube-state-metrics:8080"
            ],
            "period": "10s",
            "condition": "${kubernetes_leaderelection.leader} == true",
            "bearer_token_file": "/var/run/secrets/kubernetes.io/serviceaccount/token"
          }
        },
        {
          "enabled": true,
          "data_stream": {
            "type": "metrics",
            "dataset": "kubernetes.state_service"
          },
          "vars": {
            "add_metadata": {
              "value": true,
              "type": "bool"
            },
            "hosts": {
              "value": [
                "kube-state-metrics:8080"
              ],
              "type": "text"
            },
            "leaderelection": {
              "value": true,
              "type": "bool"
            },
            "period": {
              "value": "10s",
              "type": "text"
            },
            "node": {
              "type": "text"
            },
            "namespace": {
              "type": "text"
            },
            "bearer_token_file": {
              "value": "/var/run/secrets/kubernetes.io/serviceaccount/token",
              "type": "text"
            },
            "ssl.certificate_authorities": {
              "value": [],
              "type": "text"
            }
          },
          "id": "kubernetes/metrics-kubernetes.state_service-48db9a8e-488b-4c2a-b3b0-4dde30bdc268",
          "compiled_stream": {
            "metricsets": [
              "state_service"
            ],
            "add_metadata": true,
            "hosts": [
              "kube-state-metrics:8080"
            ],
            "period": "10s",
            "condition": "${kubernetes_leaderelection.leader} == true",
            "bearer_token_file": "/var/run/secrets/kubernetes.io/serviceaccount/token"
          }
        },
        {
          "enabled": true,
          "data_stream": {
            "type": "metrics",
            "dataset": "kubernetes.state_statefulset"
          },
          "vars": {
            "add_metadata": {
              "value": true,
              "type": "bool"
            },
            "hosts": {
              "value": [
                "kube-state-metrics:8080"
              ],
              "type": "text"
            },
            "leaderelection": {
              "value": true,
              "type": "bool"
            },
            "period": {
              "value": "10s",
              "type": "text"
            },
            "node": {
              "type": "text"
            },
            "namespace": {
              "type": "text"
            },
            "bearer_token_file": {
              "value": "/var/run/secrets/kubernetes.io/serviceaccount/token",
              "type": "text"
            },
            "ssl.certificate_authorities": {
              "value": [],
              "type": "text"
            }
          },
          "id": "kubernetes/metrics-kubernetes.state_statefulset-48db9a8e-488b-4c2a-b3b0-4dde30bdc268",
          "compiled_stream": {
            "metricsets": [
              "state_statefulset"
            ],
            "add_metadata": true,
            "hosts": [
              "kube-state-metrics:8080"
            ],
            "period": "10s",
            "condition": "${kubernetes_leaderelection.leader} == true",
            "bearer_token_file": "/var/run/secrets/kubernetes.io/serviceaccount/token"
          }
        },
        {
          "enabled": true,
          "data_stream": {
            "type": "metrics",
            "dataset": "kubernetes.state_storageclass"
          },
          "vars": {
            "add_metadata": {
              "value": true,
              "type": "bool"
            },
            "hosts": {
              "value": [
                "kube-state-metrics:8080"
              ],
              "type": "text"
            },
            "leaderelection": {
              "value": true,
              "type": "bool"
            },
            "period": {
              "value": "10s",
              "type": "text"
            },
            "node": {
              "type": "text"
            },
            "namespace": {
              "type": "text"
            },
            "bearer_token_file": {
              "value": "/var/run/secrets/kubernetes.io/serviceaccount/token",
              "type": "text"
            },
            "ssl.certificate_authorities": {
              "value": [],
              "type": "text"
            }
          },
          "id": "kubernetes/metrics-kubernetes.state_storageclass-48db9a8e-488b-4c2a-b3b0-4dde30bdc268",
          "compiled_stream": {
            "metricsets": [
              "state_storageclass"
            ],
            "add_metadata": true,
            "hosts": [
              "kube-state-metrics:8080"
            ],
            "period": "10s",
            "condition": "${kubernetes_leaderelection.leader} == true",
            "bearer_token_file": "/var/run/secrets/kubernetes.io/serviceaccount/token"
          }
        }
      ]
    },
    {
      "type": "kubernetes/metrics",
      "policy_template": "kube-apiserver",
      "enabled": true,
      "streams": [
        {
          "enabled": true,
          "data_stream": {
            "type": "metrics",
            "dataset": "kubernetes.apiserver"
          },
          "vars": {
            "bearer_token_file": {
              "value": "/var/run/secrets/kubernetes.io/serviceaccount/token",
              "type": "text"
            },
            "hosts": {
              "value": [
                "https://${env.KUBERNETES_SERVICE_HOST}:${env.KUBERNETES_SERVICE_PORT}"
              ],
              "type": "text"
            },
            "leaderelection": {
              "value": true,
              "type": "bool"
            },
            "period": {
              "value": "30s",
              "type": "text"
            },
            "ssl.certificate_authorities": {
              "value": [
                "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
              ],
              "type": "text"
            }
          },
          "id": "kubernetes/metrics-kubernetes.apiserver-48db9a8e-488b-4c2a-b3b0-4dde30bdc268",
          "compiled_stream": {
            "metricsets": [
              "apiserver"
            ],
            "hosts": [
              "https://${env.KUBERNETES_SERVICE_HOST}:${env.KUBERNETES_SERVICE_PORT}"
            ],
            "period": "30s",
            "condition": "${kubernetes_leaderelection.leader} == true",
            "bearer_token_file": "/var/run/secrets/kubernetes.io/serviceaccount/token",
            "ssl.certificate_authorities": [
              "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
            ]
          }
        }
      ]
    },
    {
      "type": "kubernetes/metrics",
      "policy_template": "kube-proxy",
      "enabled": true,
      "streams": [
        {
          "enabled": true,
          "data_stream": {
            "type": "metrics",
            "dataset": "kubernetes.proxy"
          },
          "vars": {
            "hosts": {
              "value": [
                "localhost:10249"
              ],
              "type": "text"
            },
            "period": {
              "value": "10s",
              "type": "text"
            }
          },
          "id": "kubernetes/metrics-kubernetes.proxy-48db9a8e-488b-4c2a-b3b0-4dde30bdc268",
          "compiled_stream": {
            "metricsets": [
              "proxy"
            ],
            "hosts": [
              "localhost:10249"
            ],
            "period": "10s"
          }
        }
      ]
    },
    {
      "type": "kubernetes/metrics",
      "policy_template": "kube-scheduler",
      "enabled": true,
      "streams": [
        {
          "enabled": true,
          "data_stream": {
            "type": "metrics",
            "dataset": "kubernetes.scheduler"
          },
          "vars": {
            "bearer_token_file": {
              "value": "/var/run/secrets/kubernetes.io/serviceaccount/token",
              "type": "text"
            },
            "hosts": {
              "value": [
                "https://0.0.0.0:10259"
              ],
              "type": "text"
            },
            "period": {
              "value": "10s",
              "type": "text"
            },
            "ssl.verification_mode": {
              "value": "none",
              "type": "text"
            },
            "scheduler_label_key": {
              "value": "component",
              "type": "text"
            },
            "scheduler_label_value": {
              "value": "kube-scheduler",
              "type": "text"
            }
          },
          "id": "kubernetes/metrics-kubernetes.scheduler-48db9a8e-488b-4c2a-b3b0-4dde30bdc268",
          "compiled_stream": {
            "metricsets": [
              "scheduler"
            ],
            "hosts": [
              "https://0.0.0.0:10259"
            ],
            "period": "10s",
            "bearer_token_file": "/var/run/secrets/kubernetes.io/serviceaccount/token",
            "ssl.verification_mode": "none",
            "condition": "${kubernetes.labels.component} == \\'\''kube-scheduler\\'\''"
          }
        }
      ]
    },
    {
      "type": "kubernetes/metrics",
      "policy_template": "kube-controller-manager",
      "enabled": true,
      "streams": [
        {
          "enabled": true,
          "data_stream": {
            "type": "metrics",
            "dataset": "kubernetes.controllermanager"
          },
          "vars": {
            "bearer_token_file": {
              "value": "/var/run/secrets/kubernetes.io/serviceaccount/token",
              "type": "text"
            },
            "hosts": {
              "value": [
                "https://0.0.0.0:10257"
              ],
              "type": "text"
            },
            "period": {
              "value": "10s",
              "type": "text"
            },
            "ssl.verification_mode": {
              "value": "none",
              "type": "text"
            },
            "controller_manager_label_key": {
              "value": "component",
              "type": "text"
            },
            "controller_manager_label_value": {
              "value": "kube-controller-manager",
              "type": "text"
            }
          },
          "id": "kubernetes/metrics-kubernetes.controllermanager-48db9a8e-488b-4c2a-b3b0-4dde30bdc268",
          "compiled_stream": {
            "metricsets": [
              "controllermanager"
            ],
            "hosts": [
              "https://0.0.0.0:10257"
            ],
            "period": "10s",
            "bearer_token_file": "/var/run/secrets/kubernetes.io/serviceaccount/token",
            "ssl.verification_mode": "none",
            "condition": "${kubernetes.labels.component} == \\'\''kube-controller-manager\\'\''"
          }
        }
      ]
    },
    {
      "type": "kubernetes/metrics",
      "policy_template": "events",
      "enabled": true,
      "streams": [
        {
          "enabled": true,
          "data_stream": {
            "type": "metrics",
            "dataset": "kubernetes.event"
          },
          "vars": {
            "period": {
              "value": "10s",
              "type": "text"
            },
            "add_metadata": {
              "value": true,
              "type": "bool"
            },
            "skip_older": {
              "value": true,
              "type": "bool"
            },
            "leaderelection": {
              "value": true,
              "type": "bool"
            }
          },
          "id": "kubernetes/metrics-kubernetes.event-48db9a8e-488b-4c2a-b3b0-4dde30bdc268",
          "compiled_stream": {
            "metricsets": [
              "event"
            ],
            "period": "10s",
            "add_metadata": true,
            "skip_older": true,
            "condition": "${kubernetes_leaderelection.leader} == true"
          }
        }
      ]
    },
    {
      "type": "filestream",
      "policy_template": "container-logs",
      "enabled": true,
      "streams": [
        {
          "enabled": true,
          "data_stream": {
            "type": "logs",
            "dataset": "kubernetes.container_logs"
          },
          "vars": {
            "paths": {
              "value": [
                "/var/log/containers/*${kubernetes.container.id}.log"
              ],
              "type": "text"
            },
            "symlinks": {
              "value": true,
              "type": "bool"
            },
            "containerParserStream": {
              "value": "all",
              "type": "text"
            },
            "containerParserFormat": {
              "value": "auto",
              "type": "text"
            },
            "additionalParsersConfig": {
              "value": "# - ndjson:\n#     target: json\n#     ignore_decoding_error: true\n# - multiline:\n#     type: pattern\n#     pattern: \\'\''^\\[\\'\''\n#     negate: true\n#     match: after\n",
              "type": "yaml"
            }
          },
          "id": "filestream-kubernetes.container_logs-48db9a8e-488b-4c2a-b3b0-4dde30bdc268",
          "compiled_stream": {
            "id": "kubernetes-container-logs-${kubernetes.pod.name}-${kubernetes.container.id}",
            "paths": [
              "/var/log/containers/*${kubernetes.container.id}.log"
            ],
            "prospector.scanner.symlinks": true,
            "parsers": [
              {
                "container": {
                  "stream": "all",
                  "format": "auto"
                }
              }
            ]
          }
        }
      ]
    },
    {
      "type": "filestream",
      "policy_template": "audit-logs",
      "enabled": true,
      "streams": [
        {
          "enabled": true,
          "data_stream": {
            "type": "logs",
            "dataset": "kubernetes.audit_logs"
          },
          "vars": {
            "paths": {
              "value": [
                "/var/log/kubernetes/kube-apiserver-audit.log"
              ],
              "type": "text"
            }
          },
          "id": "filestream-kubernetes.audit_logs-48db9a8e-488b-4c2a-b3b0-4dde30bdc268",
          "compiled_stream": {
            "paths": [
              "/var/log/kubernetes/kube-apiserver-audit.log"
            ],
            "exclude_files": [
              ".gz$"
            ],
            "parsers": [
              {
                "ndjson": {
                  "add_error_key": true,
                  "target": "kubernetes_audit"
                }
              }
            ],
            "processors": [
              {
                "rename": {
                  "fields": [
                    {
                      "from": "kubernetes_audit",
                      "to": "kubernetes.audit"
                    }
                  ]
                }
              },
              {
                "drop_fields": {
                  "when": {
                    "has_fields": "kubernetes.audit.responseObject"
                  },
                  "fields": [
                    "kubernetes.audit.responseObject.metadata"
                  ]
                }
              },
              {
                "drop_fields": {
                  "when": {
                    "has_fields": "kubernetes.audit.requestObject"
                  },
                  "fields": [
                    "kubernetes.audit.requestObject.metadata"
                  ]
                }
              },
              {
                "script": {
                  "lang": "javascript",
                  "id": "dedot_annotations",
                  "source": "function process(event) {\n  var audit = event.Get(\"kubernetes.audit\");\n  for (var annotation in audit[\"annotations\"]) {\n    var annotation_dedoted = annotation.replace(/\\./g,\\'\''_\\'\'')\n    event.Rename(\"kubernetes.audit.annotations.\"+annotation, \"kubernetes.audit.annotations.\"+annotation_dedoted)\n  }\n  return event;\n} function test() {\n  var event = process(new Event({ \"kubernetes\": { \"audit\": { \"annotations\": { \"authorization.k8s.io/decision\": \"allow\", \"authorization.k8s.io/reason\": \"RBAC: allowed by ClusterRoleBinding \\\"system:kube-scheduler\\\" of ClusterRole \\\"system:kube-scheduler\\\" to User \\\"system:kube-scheduler\\\"\" } } } }));\n  if (event.Get(\"kubernetes.audit.annotations.authorization_k8s_io/decision\") !== \"allow\") {\n      throw \"expected kubernetes.audit.annotations.authorization_k8s_io/decision === allow\";\n  }\n}\n"
                }
              }
            ]
          }
        }
      ]
    }
  ]
}'

[gizas]

@nchaulet Works!!!!!!! thank you so much
Closing this