Change tryAssignIPs to assign up to configured WARM_IP_TARGET #1279
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Previously tryAssignIPs would always allocate all available IPs on the ENI regardless of the number of IPs the node would want. A great showcase of this behavior is on node bootstrap. Assuming we have a node with maxIPsPerENI=30 and WARM_IP_TARGET=5, the following happens: - Node starts with single IP - tryAssignIPs allocates an additional 29 IPs - decreaseIPPool will release 29-5=24 IPs After the change this would look like: - Node starts with a single IP - tryAssignIPs allocates an additional 5 IPs So in this very common flow we cut out an entire AWS API call per node. In addition this behavior exhibits itself any time we need to scale up IPs as tryAssignIPs always attempts to over-allocate IPs -- which then just need to be released almost immediately. Fixes aws#1272
|
cc @mogren |
|
Hi @jacksontj Please kindly fill the PR template and attach unit test cases - logs showing the functionality is working should be sufficient. Appreciate your help on this. Thank you! |
nithu0115
reviewed
Nov 9, 2020
nithu0115
reviewed
Nov 9, 2020
|
Also, can you please add mock up unit test cases something similar to ones here - https://github.com/aws/amazon-vpc-cni-k8s/blob/master/pkg/awsutils/awsutils_test.go. Thank you! |
|
As @nithu0115 mentioned, we will test this change to verify if p99 pod start up time is increasing. |
haouc
reviewed
Mar 2, 2021
| // Find an ENI where we can add more IPs | ||
| eni := c.dataStore.GetENINeedsIP(c.maxIPsPerENI, c.useCustomNetworking) | ||
| if eni != nil && len(eni.IPv4Addresses) < c.maxIPsPerENI { | ||
| currentNumberOfAllocatedIPs := len(eni.IPv4Addresses) | ||
| // Try to allocate all available IPs for this ENI | ||
| err = c.awsClient.AllocIPAddresses(eni.ID, c.maxIPsPerENI-currentNumberOfAllocatedIPs) | ||
| err = c.awsClient.AllocIPAddresses(eni.ID, int(math.Min(float64(c.maxIPsPerENI-currentNumberOfAllocatedIPs), float64(toAllocate)))) |
There was a problem hiding this comment.
We have min() and max() defined in this file which can be used here without doing type conversion.
|
I cherry picked this PR on top of 1.7.10 release and tested it. The change is looks good to me. Tested it with below 2 scenarios and saw expected behavior. One thing to note was, I saw an increase in number of AssignPrivateIpAddresses API calls as we are not going optimistic behavior anymore where we previously assumed that there were pending Pods. Scenario1: Started with 1 node in the cluster and CNI configuration MINIMUM_IP_TARGET=5 and WARM_IP_TARGET=2 and launched 100 Pods. |
M00nF1sh
pushed a commit
to M00nF1sh/amazon-vpc-cni-k8s
that referenced
this pull request
Jun 7, 2021
Previously tryAssignIPs would always allocate all available IPs on the ENI regardless of the number of IPs the node would want. A great showcase of this behavior is on node bootstrap. Assuming we have a node with maxIPsPerENI=30 and WARM_IP_TARGET=5, the following happens: - Node starts with single IP - tryAssignIPs allocates an additional 29 IPs - decreaseIPPool will release 29-5=24 IPs After the change this would look like: - Node starts with a single IP - tryAssignIPs allocates an additional 5 IPs So in this very common flow we cut out an entire AWS API call per node. In addition this behavior exhibits itself any time we need to scale up IPs as tryAssignIPs always attempts to over-allocate IPs -- which then just need to be released almost immediately. Fixes aws#1272
M00nF1sh
pushed a commit
to M00nF1sh/amazon-vpc-cni-k8s
that referenced
this pull request
Jun 7, 2021
Previously tryAssignIPs would always allocate all available IPs on the ENI regardless of the number of IPs the node would want. A great showcase of this behavior is on node bootstrap. Assuming we have a node with maxIPsPerENI=30 and WARM_IP_TARGET=5, the following happens: - Node starts with single IP - tryAssignIPs allocates an additional 29 IPs - decreaseIPPool will release 29-5=24 IPs After the change this would look like: - Node starts with a single IP - tryAssignIPs allocates an additional 5 IPs So in this very common flow we cut out an entire AWS API call per node. In addition this behavior exhibits itself any time we need to scale up IPs as tryAssignIPs always attempts to over-allocate IPs -- which then just need to be released almost immediately. Fixes aws#1272
M00nF1sh
added a commit
that referenced
this pull request
Jun 7, 2021
…#1495) Previously tryAssignIPs would always allocate all available IPs on the ENI regardless of the number of IPs the node would want. A great showcase of this behavior is on node bootstrap. Assuming we have a node with maxIPsPerENI=30 and WARM_IP_TARGET=5, the following happens: - Node starts with single IP - tryAssignIPs allocates an additional 29 IPs - decreaseIPPool will release 29-5=24 IPs After the change this would look like: - Node starts with a single IP - tryAssignIPs allocates an additional 5 IPs So in this very common flow we cut out an entire AWS API call per node. In addition this behavior exhibits itself any time we need to scale up IPs as tryAssignIPs always attempts to over-allocate IPs -- which then just need to be released almost immediately. Fixes #1272 Co-authored-by: Thomas Jackson <jacksontj.89@gmail.com>
This was referenced Jun 10, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
bug / feature
Which issue does this PR fix:
#1272
What does this PR do / Why do we need it:
Previously tryAssignIPs would always allocate all available IPs on the
ENI regardless of the number of IPs the node would want. A great
showcase of this behavior is on node bootstrap. Assuming we have a node
with maxIPsPerENI=30 and WARM_IP_TARGET=5, the following happens:
After the change this would look like:
So in this very common flow we cut out an entire AWS API call per node.
In addition this behavior exhibits itself any time we need to scale up
IPs as tryAssignIPs always attempts to over-allocate IPs -- which then
just need to be released almost immediately.
If an issue # is not available please add repro steps and logs from IPAMD/CNI showing the issue:
Testing done on this change:
Automation added to e2e:
Will this break upgrades or downgrades. Has updating a running cluster been tested?:
Does this change require updates to the CNI daemonset config files to work?:
Does this PR introduce any user-facing change?:
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.