Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unexpected workload_kind named 'Replicaset' in k8s-topology #229

Closed
NeJan2020 opened this issue May 31, 2022 · 0 comments · Fixed by #230
Closed

Unexpected workload_kind named 'Replicaset' in k8s-topology #229

NeJan2020 opened this issue May 31, 2022 · 0 comments · Fixed by #230
Assignees
Labels
area/collector Issues or PRs related to agent metric collector bug Something isn't working

Comments

@NeJan2020
Copy link
Collaborator

NeJan2020 commented May 31, 2022

Describe the bug
Some k8s workload types are marked as replicaset, whose actual value is deployment.

How to reproduce?
Hard to reproduce this in a real environment. But here is an example to show why this happened.

I add some debug Info and manual sleep in collector/metadata/kubernetes/pod_watch.go and replicaset_watch.go.
Here I manually stopped the container of POD named bookdemo-d796bbdbf-jm8cv. And check what happens in k8s_watcher.
The result below shows that when the update of ReplicaSet and pod occurs synchronously, the pod may not be able to obtain the deployment information corresponding to its RS.

Debug: POD UPDATE, Name: bookdemo-d796bbdbf-jm8cv
Debug: POD Delete in cache, Name: bookdemo-d796bbdbf-jm8cv
Debug: POD Add in cache, Name: bookdemo-d796bbdbf-jm8cv
Debug: POD Add Manual sleep for 2s if pod.Name contains bookdemo

Debug: RS Update, Name: bookdemo-d796bbdbf
Debug: RS Delete in cache, Name: bookdemo-d796bbdbf
Debug: RS Add Manual sleep for 5s if rs.Name contains bookdemo

Debug: POD Add Faild to find controller for rs bookdemo-d796bbdbf

Debug: POD UPDATE, Name: bookdemo-d796bbdbf-jm8cv
Debug: POD Delete in cache, Name: bookdemo-d796bbdbf-jm8cv
Debug: POD Add in cache, Name: bookdemo-d796bbdbf-jm8cv
Debug: POD Add Manual sleep for 2s if pod.Name contains bookdemo
Debug: POD Add Faild to find controller for rs bookdemo-d796bbdbf

Debug: RS Add in cache, Name: bookdemo-d796bbdbf

Debug: RS Update, Name: bookdemo-d796bbdbf
Debug: RS Delete, Name: bookdemo-d796bbdbf
Debug: RS Add Manual sleep for 5s if rs.Name contains bookdemo
Debug: RS Add, Name: bookdemo-d796bbdbf

Add this is what I got in the '/metrics' API:

kindling_topology_request_average_duration_nanoseconds_bucket{dst_container="",dst_container_id="",dst_ip="10.99.249.214",dst_namespace="bookdemo",dst_node="",dst_pod="",dst_port="8080",dst_service="bookdemo",dst_workload_kind="replicaset",dst_workload_name="bookdemo-d796bbdbf",protocol="http",service_instance_id="10.10.103.96",service_name="cmonitor-noclusteridset",src_container="jmeterv2",src_container_id="777f49cbc498",src_ip="192.168.94.92",src_namespace="bookdemo",src_node="10.10.103.96",src_pod="jmeterv2-59d847bdd5-n69tn",src_service="jmeterv2",src_workload_kind="deployment",src_workload_name="jmeterv2",status_code="0",le="1e+07"} 2

What did you expect to see?
Whatever happens, a pod managed by any deployment should be marked as deployment in the field of workload_kind, and the field of workload_name should be the deployment's name.

@NeJan2020 NeJan2020 added bug Something isn't working area/collector Issues or PRs related to agent metric collector labels May 31, 2022
@NeJan2020 NeJan2020 self-assigned this May 31, 2022
NeJan2020 added a commit to NeJan2020/kindling that referenced this issue May 31, 2022
link KindlingProject#229

Signed-off-by: niejiangang <niejiangang@harmonycloud.cn>
dxsup pushed a commit that referenced this issue May 31, 2022
* fix: add mux durning searching replicaSet's Owner

Signed-off-by: niejiangang <niejiangang@harmonycloud.cn>

* test: add a testcase named OnAddPodWhileReplicaSetUpdating

link #229

Signed-off-by: niejiangang <niejiangang@harmonycloud.cn>

* test: a simple mistake in test case

Signed-off-by: niejiangang <niejiangang@harmonycloud.cn>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/collector Issues or PRs related to agent metric collector bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant