Adding Ip rules and route to enabling support for nodelocal DNS for Security Groups per pod

[kakarotbyte]

What would you like to be added:

Adding manual route is allowing communication
Is this a recommended approach. If So can we add this in a new release ?

Below are the replication steps:

Issue

kubectl exec -it deploy-6f796cfd45-68qhp -- cat /etc/resolv.conf
nameserver 169.254.20.10
search test.svc.cluster.local svc.cluster.local cluster.local ec2.internal
options ndots:5

kubectl exec -it deploy-6f796cfd45-68qhp -- curl -ivk google.com
* Could not resolve host: google.com
* Closing connection 0
curl: (6) Could not resolve host: google.com
command terminated with exit code 6

• When ever a pod is created a new route table is creates in below case we see 101

  ip route show table all | grep "table" | sed 's/.*\(table.*\)/\1/g' | awk '{print $2}' | sort | uniq
  101
  3
  local
  

• by default new table looks as below.

  ip route show table 101
  default via 192.168.0.1 dev vlan.eth.1
  192.168.0.1 dev vlan.eth.1 scope link
  192.168.26.147 dev vlan107abcf0bc3 scope lin
  

• Adding new rules.

  ip rule add from 169.254.20.10 to 192.168.26.147 pref 9 table 101
  ip route add local 169.254.20.10 table 101 proto kernel scope host dev nodelocaldns
  

• After adding new rules

  ip route show table 101
  default via 192.168.0.1 dev vlan.eth.1
  local 169.254.20.10 dev nodelocaldns proto kernel scope host
  192.168.0.1 dev vlan.eth.1 scope link
  192.168.26.147 dev vlan107abcf0bc3 scope link
  

• successful test

  kubectl exec -it deploy-6f796cfd45-68qhp -n test -- nslookup  google.com
  Server:		169.254.20.10
  Address:	169.254.20.10#53
  
  Non-authoritative answer:
  Name:	google.com
  Address: 172.217.12.238
  Name:	google.com
  Address: 2607:f8b0:4004:83f::200e
  

Why is this needed:
Based on the issue 1384 currently NodeLocal DNS doesnt work with Security group for pods. This ask will allow the communication to be supported.

[achevuru]

@kakarotbyte We're working on a long term solution for this use case. We will provide an update soon.

[haouc]

@kakarotbyte, just a quick update. A engineer in our team has made some progress on this support. We are having meeting this week to finalize the design and implementations. We will provide the update for further timelines.

[kakarotbyte]

Thank you for update @haouc

[nip572]

@haouc any updates on this? There are alternates that we are developing to get Pod Sgs to work with Node Local DNS so if the release is near, we can halt our work around this.

[haouc]

@nip572 sorry for the late response. We are working on the feature to support this use case now. I will provide updates soon.

[virtualdom]

Hi @haouc! Appreciate the work you all are doing on this as my team is also really interested in a resolution here. Are there any updates on this?

[M00nF1sh]

Hi, we are working on this and ETA is Q1 2022.

The resolution is to provide a new SGPP mode where users need to opt-in by set an environment variable. Once opted-in, the SGPP pods will behaves same(routing rules) as normal pod(based on secondary IPs and normal ENI), except it's based on a dedicated branch ENI.

[jayanthvn]

v1.11.0 is released. Closing the issue now. Please do reach out for any follow ups.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.