Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix hostport not processed in k8sprocessor #219

Merged
merged 6 commits into from
May 13, 2022

Conversation

dxsup
Copy link
Member

@dxsup dxsup commented May 12, 2022

Description

  1. Record containers using hostPort as a new map used for searching pod information when the destination is in hostIp:hostPort format.
  2. The method GetDNATTuple in the conntracker module returned both DNAT and SNAT connections before. This PR fixes this bug. This bug could bring unexpected dst_ip and dst_port labels when we get SNAT connections from the method GetDNATTuple incorrectly.

Motivation and Context

What we expect to see for the metric labels of hostPort scenario is:

  1. dst_ip is the pod's IP.
  2. dst_port is the container's port that is exposed to the hostport.
  3. dst_service should be hostIP:hostPort which represents how the pod is called.

The scenario of containers using hostPort was not considered before, so when a client calls a pod using HostIP:HostPort, the processed data didn't contain the information of the pod. This PR resolves such a problem by recording the hostPorts as a new map and looking up the map for the container's metadata.

GetDNATTuple bug

tcp      6 83 TIME_WAIT 
(Send)src=192.168.94.121 dst=10.10.101.124 sport=33532 dport=18080 
(Reply)src=10.10.101.124 dst=10.10.103.96 sport=18080 dport=33532 [ASSURED] 
mark=0 secctx=system_u:object_r:unlabeled_t:s0 use=1

This is one record shown by executing conntrack -L.

  • 192.168.94.121 is the IP of a Pod named A.
  • 10.10.101.124 is the host IP of another Pod named B.
  • 18080 is the hostPort of the Pod B.
  • 10.10.103.96 is the host IP of the Pod A.

Since Send.dst == Reply.src and Send.dport == Reply.sport, we know there is no DNAT happened. But because Send.src != Reply.Dst, SNAT happened. This record was returned by the method GetDNATTuple before, which resulted in incorrect value in dnat_ip and dnat_port fields with the following piece of codes.

	if nil != mps.natTuple && mps.responses != nil {
		labels.UpdateAddStringValue(constlabels.DnatIp, mps.natTuple.ReplSrcIP.String())
		labels.UpdateAddIntValue(constlabels.DnatPort, int64(mps.natTuple.ReplSrcPort))
	}

Considering that we replace dst_ip and dst_port with dnat_ip and dnat_port in the adapters of otelexporter, if the fields dnat_ip and dnat_port are incorrect, the same dst_ip and dst_port will be.

func replaceDstIpOrDstPortByDNat() adjustFunctions {
	return adjustFunctions{
		adjustAttrMaps: func(labels *model.AttributeMap, attributeMap *model.AttributeMap) *model.AttributeMap {
			dNatIp := labels.GetStringValue(constlabels.DnatIp)
			dNatPort := labels.GetIntValue(constlabels.DnatPort)
			if dNatIp == "" || dNatPort < 1 {
				return attributeMap
			} else {
				attributeMap.AddStringValue(constlabels.DstIp, dNatIp)
				attributeMap.AddIntValue(constlabels.DstPort, dNatPort)
				return attributeMap
			}
		},
        ....
}

Until here it looks like the problem is not serious, because if no DNAT happened, dnat_ip should always equal to dst_ip. There is no effect even if we replace dst_ip with the same dnat_ip. But when it comes to hostPort, everything changes.

When a pod is called via hostIP:hostPort, the original dst_ip is hostIP and the original dst_port is hostPort. In this PR, we replace dst_ip(hostIP) and dst_port(hostPort) with its pod's IP and container's port. If they are replaced by dnat_ip and dnat_port again, the final result will be not expected.

Daxin Wang added 6 commits May 12, 2022 18:04
Signed-off-by: Daxin Wang <daxinwang@harmonycloud.cn>
Signed-off-by: Daxin Wang <daxinwang@harmonycloud.cn>
Signed-off-by: Daxin Wang <daxinwang@harmonycloud.cn>
Signed-off-by: Daxin Wang <daxinwang@harmonycloud.cn>
Signed-off-by: Daxin Wang <daxinwang@harmonycloud.cn>
Signed-off-by: Daxin Wang <daxinwang@harmonycloud.cn>
@dxsup dxsup requested a review from NeJan2020 May 12, 2022 11:16
Copy link
Collaborator

@NeJan2020 NeJan2020 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dxsup dxsup merged commit eb9dfba into KindlingProject:main May 13, 2022
@dxsup dxsup deleted the fix/hostport-not-found branch May 13, 2022 07:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants