How to disable debug level for cni-metrics?

[csantanapr]

What happened:

The cni-metric-helper pod is logging to much information, is dumping all metrics to logs and this is a cost and operation overhead.

This is the default behaivor, I think we should change the log level to be as expected in production environment error only, if someone is trying to debug or develop then they can change it to high verbose level like debug

I installed the cni-metric-helper using the instructions here https://docs.aws.amazon.com/eks/latest/userguide/cni-metrics-helper.html

[jdn5126]

@csantanapr this is a good call out. Can you share which version of cni-metrics-helper you are using and some example logs that you think should be suppressed? I can dig further, but it seems like cni-metrics-helper respects the AWS_VPC_K8S_CNI_LOGLEVEL environment variable (default is Debug as you mentioned).

[csantanapr]

Hi @jdn5126

The version is latest version 1.12.1
The logs are too much to include here, but I'm referring to the metrics content like this:

{"level":"debug","ts":"2023-01-13T01:19:16.274Z","caller":"metrics/metrics.go:382","msg":"cni-metrics text output: # HELP awscni_add_ip_req_count The number of add IP address requests\n# TYPE awscni_add_ip_req_count counter\nawscni_add_ip_req_count 2\n# HELP awscni_assigned_ip_addresses The number of IP addresses assigned to pods\n# TYPE awscni_assigned_ip_addresses gauge\nawscni_assigned_ip_addresses 1\n# HELP awscni_assigned_ip_per_cidr The total number of IP addresses assigned per cidr\n# TYPE awscni_assigned_ip_per_cidr gauge\nawscni_assigned_ip_per_cidr{cidr=\"100.64.1.16/28\"} 1\n# HELP awscni_aws_api_latency_ms AWS API call latency in ms\n# TYPE awscni_aws_api_latency_ms summary\nawscni_aws_api_latency_ms_sum{api=\"AssignPrivateIpAddresses\",error=\"false\",stat
...
...

Thanks for letting me know I can use AWS_VPC_K8S_CNI_LOGLEVEL but from the README.md is not very clear that this variable is application to metrics helper container also. Should we update the docs to state which env variables are only applicable to metric server vs. ipamd deamon set?

Should we update the yaml and helm chart for metric server to add this variable and set it to other than DEBUG ? like ERROR

[jdn5126]

I see, yeah https://github.com/aws/amazon-vpc-cni-k8s/blob/master/cmd/cni-metrics-helper/metrics/cni_metrics.go#L196 is definitely too verbose by default. INFO is probably a better default, and customers can set ERROR in the helm chart if they truly only want to see error conditions.

I agree that AWS_VPC_K8S_CNI_LOGLEVEL should be documented in the README and its default added to https://github.com/aws/amazon-vpc-cni-k8s/blob/master/charts/cni-metrics-helper/values.yaml

I'll mark this as a good first issue since setting the environment variable is a workaround. I'll take up the effort if no one can get to it before me.

[hkhelif]

@jdn5126 you mentioned that the cni-metrics-helper respects the AWS_VPC_K8S_CNI_LOGLEVEL environment variable, would changing the defaultLogLevel here to INFO fix this issue?

Then the users can set the desired log level by setting it from the environment variable.

I have tested the suggested changes and I was able to only see INFO logs, then I set AWS_VPC_K8S_CNI_LOGLEVEL to DEBUG and I was able to get to the desired output.

Sorry if my questions or suggested implementation are silly, I am new to this project and new to contributing to open source in general 😄

[jdn5126]

@hkhelif you are right on track, but just setting the environment variable to DEBUG or INFO should have the same effect without requiring a change to the source code. In general, we are very cautious about changing the default value of anything, but for CNI metrics helper, I think it is reasonable.

[hkhelif]

@jdn5126 I see that there is already an open PR for this issue, should I open a new one, or it is not required anymore as #2226 already archives the same goal?

[jdn5126]

@jdn5126 I see that there is already an open PR for this issue, should I open a new one, or it is not required anymore as #2226 already archives the same goal?

Yeah, #2226 will take care of this issue. If you are interested in contributing, anything here that is unassigned is fair game

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.