Configure log levels for calico-node

[cheeryfella]

Would like to be able to configure the log level of calico nodes deployed via the tigera operator

The relevant environment variables appear to be:

• CALICO_STARTUP_LOGLEVEL
• BGP_LOGSEVERITYSCREEN
• FELIX_LOGSEVERITYSCREEN

Expected Behavior

Be able to set these values on the calico-node daemonset for the container environment variables via the tigera-operator helm chart.

Current Behavior

based on the code in https://github.com/tigera/operator/blob/master/pkg/render/node.go it would appear non of these are available to be set.

Context

Unable to change the log level of the calico-node

[lwr20]

In an operator install, you set these config parameters using the felixconfiguration object.

https://projectcalico.docs.tigera.io/reference/resources/felixconfig

[caseydavenport]

CALICO_STARTUP_LOGLEVEL

This one is not configurable by the operator to my knowledge, so requires some code work.

The others are available through the FelixConfiguration and BGPConfiguration APIs.

[Chili-Man]

@caseydavenport I tried @lwr20's suggestion by creating a FelixConfiguration like below:

---
apiVersion: projectcalico.org/v3
kind: FelixConfiguration
metadata:
  name: log-level

spec:
  logSeverityFile: Error
  logSeverityScreen: Error
  logSeveritySys: Error

However, I don't observe the log level being reduced on the calico-node daemonset. It works when I directly modify the existing default resource, but I don't want to particularly do that since I want to manage the configuration through Helm as well. How does it work alongside the default configuration?

When I do have Helm try to manage the existing default, I end up with the following error (which is expected):

Helm upgrade failed: rendered manifests contain a resource that already exists. Unable to continue with update: FelixConfiguration "default" in namespace "" exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: missing key "meta.helm.sh/release-name": must be set to "calico-policies"; annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "tigera-operator"

I understand that I can add the helm labels manually, so that it works, but I'd rather have this all automated out if possible.

[lwr20]

https://projectcalico.docs.tigera.io/reference/resources/felixconfig

Calico automatically creates a resource named default containing the global default configuration settings for Felix.

The resources with the name node.<nodename> contain the node-specific overrides, and will be applied to the node .

So the only names you should be using for Felixconfigurations are default - for configs that apply to all nodes or node.<nodename> for configs that apply to only a single node.
Your log-level Felixconfig is going to be ignored.

[Chili-Man]

thanks at @lwr20 , thats what I figured. it's unfortunate that it requires manual intervention in order to configure the felix configuration automatically with Helm

[Chili-Man]

@caseydavenport Even if i try to adopt the default FelixConfiguration through helm as specified through here helm/helm#7649 , it still doesn't update any changes that I made. I'm assuming its probably related to how Calico handles the CRDs. Looks like there's no way to adopt into Helm or any gitops system like flux and thus the only way to actually configure those log settings right now is to manually edit the default FelixConfiguration 😞

[lwr20]

I'm surprised that no gitops system can do the equivalent of a kubectl apply. Is it that the gitops platforms assume all applications can be installed in one helm or kubectl command?

Attempting to think of workarounds: can you treat the felix config as a separate "application" for applying after Calico install has completed?

The Calico Openshift install has the concept of a configmap where you specify all the calico resources you want to configure at install time, and it applies them all repeatedly until its successful. Perhaps that approach would work here too?

[Chili-Man]

Indeed, it is rather disappointing, but understandable from the goals of gitops. I think gitops assumes that it's managing the entire lifecycle of the resources, it generally doesn't support adopting existing resources that were created outside of it.

I actually do treat the felix config as a separate application, in particular, its a separate helm chart that gets installed only after the upstream calico helm chart has been successfully installed. As I mentioned, I tried to get Helm to adopt the existing resource per Helm's instructions, but it does not work (which I assume is because of how Calico treats CRDs, namely that it has a separate calico api server to convert / validate the calico crds) . The only other way supported through Flux, is to use a kustomization, but kustomize does not support adopting existing resources, it only works on resources it creates from the get go 😞

I'm not familiar with openshift in general, but as I far as I know there's no equivalent feature in the AWS EKS cluster that I'm using.

[lwr20]

which I assume is because of how Calico treats CRDs, namely that it has a separate calico api server to convert / validate the calico crds

OK, so its not really a good idea, but some (not all) of the v3 apis are backed by a CRD v1 object. I guess as a workaround you could apply the v1 object directly. The downside of that approach is that there is no validation for the objects that get applied, and any object that gets applied with an invalid field will be ignored by Calico, leading to hard to understand problems later.

[Chili-Man]

heh, yeah I actually tried that a while ago when I was trying to install everything through a single helm chart; I opened up a ticket about it here #6406, which ended up prompting this discussion here #6412. The short answer is that trying to use the V1 objects won't work unfortunately.