Karmada-controller-manager stale state

[LavredisG]

Please provide an in-depth description of the question you have:
I started with a setup of 2 kind clusters, karmada-host and member1. The former was used both as the host were Karmada was installed, as well as a member cluster. So I had 2 member clusters joined. I also had set kube-prometheus-stack on karmada-host in order to retrieve karmada metrics. More specifically I was scraping karmada-controller-manager. While inspecting the cluster-node-number metric I was getting the expected result.

However, once I deleted member1 cluster, karmada-controller-manager returned info about 2 clusters instead of 1 when querried.

What do you think about this question?:
Is that expected behavior? Shouldn't karmada-controller-manager observe that 1 cluster has been deleted so that it shouldn't retrieve info about it anymore?

Environment:

• Karmada version: v1.12.0-alpha.1-36-g13df63fa4
• Kubernetes version: v1.31.0
• kubectl: v1.29.0

[RainbowMango]

While inspecting the cluster-node-number metric I was getting the expected result.

Firstly, I don't remember having that metric emitted by the karmada-controller-manager.
Are you sure that's from karmada-controller-manager?

However, once I deleted member1 cluster,

How do you delete the cluster? By karmadactl unjoin or just removed the kind cluster, by a command like kind delete clusters --name=member1?

[LavredisG]

I am referring to this one:

I first unjoined the cluster and then deleted it from kind. A simple workaround is to delete the controller pod, so that when it gets recreated it picks up the correct state, but I don't know if that's the expected way to do it.

[RainbowMango]

Oh, I see. That was indeed reported from Karmada.
We didn't clean up the gauge after the cluster was gone.

A simple workaround is to delete the controller pod, so that when it gets recreated it picks up the correct state, but I don't know if that's the expected way to do it.

I think it's not the expected behavior. We need more explore how to fix it.

By the way, how important is the metric to you?

[RainbowMango]

By the way, let me know if you are going to explore this?

[LavredisG]

My goal is to create a custom scheduler for karmada, utilizing resource/power/network metrics, but I have not concluded which I will use, so at this point I was just trying to set up connectivity from karmada metrics to Prometheus.

By exploring you mean if I will try to fix it?

[RainbowMango]

Yeah, I mean if you like you can dig into this issue, find the root cause, then try to fix it :)

[LavredisG]

Yeah, I mean if you like you can dig into this issue, find the root cause, then try to fix it :)

Unfortanutely, I don't think I have enough time to fix it in the foreseeable future :/
I would appreciate to be informed on any updates on this though!

[RainbowMango]

That's OK.
Given this issue is part of #5843, let's track it by that issue.
/close
Duplicate of #5843

[karmada-bot]

@RainbowMango: Closing this issue.

In response to this:

That's OK.
Given this issue is part of #5843, let's track it by that issue.
/close
Duplicate of #5843

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.