[Question] How should tunnel-server be deployed

[adamzhoul]

What happened:

tunnel-server is deployed as deployment and scheduled to the node which is not the same as apiServer.

What you expected to happen:

as far as I know, it should change iptables, to redirect apiServer traffic to self when targeting edge kublet.
if they are not on the same node, how to do this.

How to reproduce it (as minimally and precisely as possible):

check the template:

openyurt/pkg/yurtctl/constants/yurt-tunnel-server-tmpl.go

Line 191 in 47a88ea

nodeSelector:

others
/kind question

[rambohe-ch]

@adamzhoul Welcome you to apply OpenYurt community member. you can add me as sponsor and apply example is here: openyurtio/community#23

[rambohe-ch]

What happened:

tunnel-server is deployed as deployment and scheduled to the node which is not the same as apiServer.

What you expected to happen:

as far as I know, it should change iptables, to redirect apiServer traffic to self when targeting edge kublet.
if they are not on the same node, how to do this.

How to reproduce it (as minimally and precisely as possible):

check the template:

openyurt/pkg/yurtctl/constants/yurt-tunnel-server-tmpl.go

Line 191 in 47a88ea

nodeSelector:

others
/kind question

@adamzhoul Yurt-tunnel-server support dns mode to forward traffic from kube-apiserver to edge kubelet.

@DrmagicE Would you can give more detail configuration of yurt-tunnel-server to support dns mode.

[adamzhoul]

simplify my problem is :
if Yurt-tunnel-server is not deployed on the same node as the apiServer
Whatever iptables operation it does, can't affect apiServer outbound traffic.

isn't that?

[rambohe-ch]

simplify my problem is :
if Yurt-tunnel-server is not deployed on the same node as the apiServer
Whatever iptables operation it does, can't affect apiServer outbound traffic.

isn't that?

@adamzhoul Thanks for your feedback.

1. Iptables mode is not suitable for this scenario.
2. we need to configure kube-apiserver(----kubelet-preferred-address-types) to use nodeName:Port to access edge nodes, so dns resolution will be triggered.
3. and yurt-tunnel-server support dns mode for resolve nodeName to the address of yurt-tunnel-server. so the kube-apiserver outbound traffic will be forwarded to yurt-tunnel-server.

[adamzhoul]

hi @rambohe-ch

search k8s code found. here:

I assume you mean type: Hostname
and, try to figure out the workflow.

apiServer part

1. configure kube-apiserver --kubelet-preferred-address-types=Hostname

2. apiServer try to call kubelet, find kublete node ip by func : GetPreferredNodeAddress

3. get the name from node.status.address type=Hostname , get node hostname

4. DNS resolve hostname to IP ( IP finally resolved to yurt-tunnel-server-part svc IP)

yurt-tunnel-server part

1. run coreDNSRecordController, start nodeInformer
2. handle event addNode(obj interface{})
3. update node self svc ip map into configmap yurt-tunnel-nodes

Questions

so , I still have two questions:

1. How does the configmap effect coreDNS?
2. I see iptables dir in yurt-tunnel-server, what is it for?

[rambohe-ch]

hi @rambohe-ch

search k8s code found. here:

I assume you mean type: Hostname
and, try to figure out the workflow.

apiServer part

1. configure kube-apiserver --kubelet-preferred-address-types=Hostname
2. apiServer try to call kubelet, find kublete node ip by func : GetPreferredNodeAddress

1. get the name from node.status.address type=Hostname , get node hostname
2. DNS resolve hostname to IP ( IP finally resolved to yurt-tunnel-server-part svc IP)

yurt-tunnel-server part

1. run coreDNSRecordController, start nodeInformer
2. handle event addNode(obj interface{})
3. update node self svc ip map into configmap yurt-tunnel-nodes

Questions

so , I still have two questions:

1. How does the configmap effect coreDNS?

--> I'm sorry we do not have tutorial at present, about coredns setting you can reference this pull request: #270

By the way, i have raised an issue that look for help for a tutorial about yurt-tunnel-server dns mode: #417, if you have interested in it, you can take over this issue.

2. I see iptables dir in yurt-tunnel-server, what is it for?

--> Both iptables mode and dns mode of yurt-tunnel-server can be supported at the same time.

[adamzhoul]

@rambohe-ch thanks for answering.

I can try to help with the DNS mod doc once figure out each part of the DNS mode.

[rambohe-ch]

@rambohe-ch thanks for answering.

I can try to help with the DNS mod doc once figure out each part of the DNS mode.

@adamzhoul Welcome you to apply OpenYurt community member, here is an example: openyurtio/community#23

[DrmagicE]

@adamzhoul Hi, here is my experience on setting yurt-tunnel DNS mode, it may not be the best practice, but it works for me.

1. Use nodepool to separate cloud node and edge node. (This is for service topology)
2. Deploy Yurthub manually on the cloud-side and also configure the kubelet to connect to Yurthub.
3. Use daemonset to deploy coredns.
4. Edit kube-proxy configmap, and re-deploy kube-proxy

# kubectl edit cm kube-proxy -n kube-system
apiVersion: v1
data:
  config.conf: |-
    apiVersion: kubeproxy.config.k8s.io/v1alpha1
    bindAddress: 0.0.0.0
    featureGates: # ①
      EndpointSliceProxying: true
    clientConnection:
      acceptContentTypes: ""
      burst: 0
      contentType: ""
      # ② kubeconfig: /var/lib/kube-proxy/kubeconfig.conf 
      qps: 0
      ...
# delete pods -l k8s-app=kube-proxy -n kube-system    

①：Add EndpointSliceProxying featurGates.
②: Remove kubeconfig so that the kube-proxy can use yurthub to forward traffic.
5. Edit coredns daemonset YAML: #270
6. Add topology annotation to kube-dns service:

apiVersion: v1
kind: Service
metadata:
  annotations:
    openyurt.io/topologyKeys: openyurt.io/nodepool

7. Edit kube-apiserver manifest:

#  vi /etc/kubernetes/manifests/kube-apiserver.yaml
   - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
    - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
    - --kubelet-preferred-address-types=Hostname,InternalIP,ExternalIP  # ①
    - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
    - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
...
dnsPolicy: ClusterFirstWithHostNet # ②

① Put "Hostname" in the first place.
② Configure dnsPolicy to ClusterFirstWithHostNet，which enable apiserver to make use of coredns.

Hope it helps.

[adamzhoul]

thanks @DrmagicE it helps a lot.

The dns controller will generate a ConfigMap named yurt-tunnel-nodes in the kube-system namespace.
You need to manually config CoreDNS to use it according to the steps like the following:

check the pr. so we do have to manually config to volume config to coreDNS.

and, the final target is to make all this automatic, no manual config, right?

[adamzhoul]

@rambohe-ch thanks for answering.
I can try to help with the DNS mod doc once figure out each part of the DNS mode.

@adamzhoul Welcome you to apply OpenYurt community member, here is an example: openyurtio/community#23

thanks. working on it.