Update documentation for PD (#1539)

charts/aws-vpc-cni/test.yaml

@@ -41,7 +41,9 @@ env:
   DISABLE_INTROSPECTION: "false"
   DISABLE_METRICS: "false"
   ENABLE_POD_ENI: "false"
+  ENABLE_PREFIX_DELEGATION: "false"
   WARM_ENI_TARGET: "1"
+  WARM_PREFIX_TARGET: "1"
 
 # this flag enables you to use the match label that was present in the original daemonset deployed by EKS
 # You can then annotate and label the original aws-node resources and 'adopt' them into a helm release

charts/aws-vpc-cni/values.yaml

@@ -43,7 +43,9 @@ env:
   DISABLE_INTROSPECTION: "false"
   DISABLE_METRICS: "false"
   ENABLE_POD_ENI: "false"
+  ENABLE_PREFIX_DELEGATION: "false"
   WARM_ENI_TARGET: "1"
+  WARM_PREFIX_TARGET: "1"
 
 # this flag enables you to use the match label that was present in the original daemonset deployed by EKS
 # You can then annotate and label the original aws-node resources and 'adopt' them into a helm release

config/master/manifests.jsonnet

@@ -181,6 +181,7 @@ local awsnode = {
                   },
                 },
                 WARM_ENI_TARGET: "1",
+                WARM_PREFIX_TARGET: "1",
               },
               env: [
                 {name: kv[0]} + if std.isObject(kv[1]) then kv[1] else {value: kv[1]}

docs/prefix-and-ip-target.md

@@ -1,10 +1,10 @@
 ## `WARM_PREFIX_TARGET`, `WARM_IP_TARGET` and `MINIMUM_IP_TARGET`
 
-IPAMD will start allocating (/28) prefixes to the ENIs with `ENABLE_PREFIX_DELEGATION` set to `true`. By default IPAMD will allocate 1 prefix for the allocated ENI but based on the need the number of prefixes to be held in warm pool can be controlled by setting `WARM_PREFIX_TARGET`, `WARM_IP_TARGET` and `MINIMUM_IP_TARGET` environment variables. 
+IPAMD will start allocating (/28) prefixes to the ENIs with `ENABLE_PREFIX_DELEGATION` set to `true`. Setting either `WARM_PREFIX_TARGET` or both `WARM_IP_TARGET` and `MINIMUM_IP_TARGET` environment variables to zero is not supported with prefix delegation enabled. If we set either of these to zero, it will significantly slow IP assignment to the pods as IPAMD will not maintain any additional prefixes in it's warm pool and the pod startup latency becomes even more pronounced if IPAMD needs to allocate and attach a new ENI (and wait for IMDS sync) before it can assign IPs to new pods. So, we decided against supporting this combination.
 
 `WARM_IP_TARGET` and `MINIMUM_IP_TARGET` if set will override `WARM_PREFIX_TARGET`. `WARM_PREFIX_TARGET` will allocate one full (/28) prefix even if a single IP is consumed with the existing prefix. If the ENI has no space to allocate a prefix then a new ENI will be created. So make sure to use this on need basis i.e, if pod density is high since this will be carved out of the ENIs subnet. `WARM_IP_TARGET` and `MINIUM_IP_TARGET` give more fine grained control on the number of IPs but if existing prefixes are not sufficient to maintain the warm pool then IPAMD will allocate more prefixes to the existing ENI or create a new ENI if the existing ENIs are running out of prefixes.
 
-When a new ENI is allocated, IPAMD will allocate either 1 prefix or number of prefixes needed to maintain the `WARM_PREFIX_TARGET`, `WARM_IP_TARGET` and `MINIMUM_IP_TARGET` setting. This is done to avoid extra EC2 calls to either allocate more prefixes or free extra prefixes on ENI bring up.
+When a new ENI is allocated, IPAMD will determine the number of prefixes needed to maintain the `WARM_PREFIX_TARGET` or `WARM_IP_TARGET` and `MINIMUM_IP_TARGET` setting and then those many prefixes will be allocated. This is done to avoid extra EC2 calls to allocate more prefixes and later free extra prefixes on ENI bring up. New ENI will only be allocated after we exhaust all the prefixes allocated to existing ENIs.
 
 
 Some example cases: