Network Policy Rule on Instance Metadata Endpoint

[atilsensalduz]

Hi,

Despite having the following rules in the network policy definition, I've noticed DENY logs. Is there something specific to consider when utilizing the instance metadata endpoint?

Rule:

   # Allow access to instance metadata endpoint
    - ports:
      - port: 80
        protocol: TCP
      to:
      - ipBlock:
          cidr: 169.254.169.254/32
    # Allow internet access
    - ports:
      - port: 443
        protocol: TCP
      to:
      - ipBlock: 
          cidr: 0.0.0.0/0
          # Exclude traffic to Kubernetes service IPs and pods
          except:
            - 10.0.0.0/8
            - 172.16.0.0/12
            - 192.168.0.0/16
            - 169.254.169.254/32

Here's an example log entry:

Node: ip-10-0-100-44.ec2.internal;SIP: 10.0.115.28;SPORT: 37758;DIP: 169.254.169.254;DPORT: 80;PROTOCOL: TCP;PolicyVerdict: DENY

Any insights would be appreciated

[achevuru]

@atilsensalduz Does the Source IP in the above log map to a pod to which the policy is applied? Can you do a describe on corresponding policyEndpoint resource and check if 10.0.115.28 is under the list of pod selector endpoints?

[jayanthvn]

What is your node agent version? Since I also see you are using except and there was a known issue #92 which was fixed with v1.0.4. VPC CNI v1.15.1 is released on GitHub: https://github.com/aws/amazon-vpc-cni-k8s/releases/tag/v1.15.1 and contains the agent v1.0.4 image tag.

[atilsensalduz]

Hey guys,

@achevuru, I checked. When I describe policyendpoints of the network policy, I can see the IP of the pod in the logs.

@jayanthvn, I am using the latest version of vpc-cni (v1.15.1) and the agent version is v1.0.4.

Thanks for looking into this with me!

[jdn5126]

@atilsensalduz not sure if your issue was resolved, but just noting that Network Policy agent v1.0.5 is now available with the latest VPC CNI version, v1.15.3

[atilsensalduz]

Hi @jdn5126 , I've updated the vpc-cni version and retested the connectivity, but it's still blocked. Interestingly, when I remove 169.254.169.254/32 from the 'except' block of the 'allow internet' rule, I can access the metadata endpoint. I expected the network policy to allow it with an explicit rule. Any ideas?

[achevuru]

@atilsensalduz Why do we have IMDS IP under except block of the second block?

[atilsensalduz]

I'm using Helm to generate network policies, and one of the rules in the network policy template restricts internet access, which inadvertently prevents access to the metadata endpoint. To address this, I've added a separate rule for applications that genuinely require access to the metadata endpoint. 😄

[achevuru]

Ok, I'm referring to IMDS IP being specified both in the allow and except blocks above (i.e.,) the below to block is allowing access to IMDS endpoint over TCP and port 80. So, IMDS endpoint is expected to be allowed for all pods for that port and protocol combo and all other port and protocols will be denied for IMDS anyways. So, we don't need to include IMDS again in the except block for port 443 (in the second to block). Deny always wins...

    - ports:
      - port: 80
        protocol: TCP
      to:
      - ipBlock:
          cidr: 169.254.169.254/32

[atilsensalduz]

thanks a bunch @achevuru

"Deny always wins..." I like this quote. I might even consider getting it tattooed! 😄