Skip to content

Commit

Permalink
Add support for a 'no manage' tag (aws#726)
Browse files Browse the repository at this point in the history 
* Add support for a 'no manage' tag

This tag results in an ENI being unmanaged by the ipamd plugin. This
allows someone to create an ENI, associate it with an instance, and then
use eks per normal while also using that ENI for whatever.

* Update tests for new changes

* Remove 'unmanaged eni' metric

* Update readme with docs about CLUSTER_NAME and no_manage

* Correctly indent tag sub-headers

* Remove trailing '.'

* Account for unmanaged ENIs in allocation check
  • Loading branch information
@euank
euank authored and Claes Mogren committed Dec 15, 2019
1 parent fe94613 commit 57eab64
Show file tree
Hide file tree
Showing 12 changed files with 242 additions and 103 deletions.
49 changes: 46 additions & 3 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ The default manifest expects `--cni-conf-dir=/etc/cni/net.d` and `--cni-bin-dir=

L-IPAM requires following [IAM policy](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html):

```
```
{
"Effect": "Allow",
"Action": [
Expand Down Expand Up @@ -375,10 +375,53 @@ Default: `{}`

Example values: `{"tag_key": "tag_val"}`

Metadata applied to ENI help you categorize and organize your resources for billing or other purposes. Each tag consists of a custom-defined key and an optional value. Tag keys can have a maximum character length of 128 characters. Tag values can have a maximum length of 256 characters. These tags will be added to all ENIs on the host.
Metadata applied to ENI help you categorize and organize your resources for billing or other purposes. Each tag consists of a custom-defined key and an optional value. Tag keys can have a maximum character length of 128 characters. Tag values can have a maximum length of 256 characters. These tags will be added to all ENIs on the host.

Important: Custom tags should not contain `k8s.amazonaws.com` prefix as it is reserved. If the tag has `k8s.amazonaws.com` string, tag addition will ignored.

---

`CLUSTER_NAME`

Type: String

Default: `""`

Specifies the cluster name to tag allocated ENIs with. See the "Cluster Name tag" section below.

### ENI tags related to Allocation

This plugin interacts with the following tags on ENIs:

* `cluster.k8s.amazonaws.com/name`
* `node.k8s.amazonaws.com/instance_id`
* `node.k8s.amazonaws.com/no_manage`

#### Cluster Name tag

The tag `cluster.k8s.amazonaws.com/name` will be set to the cluster name of the
aws-node daemonset which created the ENI.

#### Instance ID tag

The tag `node.k8s.amazonaws.com/instance_id` will be set to the instance ID of
the aws-node instance that allocated this ENI.

#### No Manage tag

The tag `node.k8s.amazonaws.com/no_manage` is read by the aws-node daemonset to
determine whether an ENI attached to the machine should not be configured or
used for private IPs.

This tag is not set by the cni plugin itself, but rather may be set by a user
to indicate that an ENI is intended for host networking pods, or for some other
process unrelated to Kubernetes.

*Note*: Attaching an ENI with the `no_manage` tag will result in an incorrect
value for the Kubelet's `--max-pods` configuration option. Consider also
updating the `MAX_ENI` and `--max-pods` configuration options on this plugin
and the kubelet respectively if you are making use of this tag.

### Notes

`L-IPAMD`(aws-node daemonSet) running on every worker node requires access to kubernetes API server. If it can **not** reach
Expand All @@ -405,4 +448,4 @@ instructions [here](https://aws.amazon.com/security/vulnerability-reporting/) or

## Contributing

[See CONTRIBUTING.md](./CONTRIBUTING.md)
[See CONTRIBUTING.md](./CONTRIBUTING.md)
58 changes: 42 additions & 16 deletions ipamd/ipamd.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,10 @@ const (
// This environment is used to specify whether Pods need to use a security group and subnet defined in an ENIConfig CRD.
// When it is NOT set or set to false, ipamd will use primary interface security group and subnet for Pod network.
envCustomNetworkCfg = "AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG"

// eniNoManageTagKey is the tag that may be set on an ENI to indicate ipamd
// should not manage it in any form.
eniNoManageTagKey = "node.k8s.amazonaws.com/no_manage"
)

var (
Expand All @@ -127,7 +131,7 @@ var (
enisMax = prometheus.NewGauge(
prometheus.GaugeOpts{
Name: "awscni_eni_max",
Help: "The maximum number of ENIs that can be attached to the instance",
Help: "The maximum number of ENIs that can be attached to the instance, accounting for unmanaged ENIs",
},
)
ipMax = prometheus.NewGauge(
Expand Down Expand Up @@ -170,6 +174,7 @@ type IPAMContext struct {
networkClient networkutils.NetworkAPIs
maxIPsPerENI int
maxENI int
unmanagedENI int
warmENITarget int
warmIPTarget int
minimumIPTarget int
Expand Down Expand Up @@ -271,25 +276,25 @@ func (c *IPAMContext) nodeInit() error {

log.Debugf("Start node init")

c.maxENI, err = c.getMaxENI()
allENIs, err := c.awsClient.GetAttachedENIs()
if err != nil {
log.Error("Failed to retrieve ENI info")
return errors.New("ipamd init: failed to retrieve attached ENIs info")
}
enis, numUnmanaged := filterUnmanagedENIs(allENIs)
nodeMaxENI, err := c.getMaxENI()
if err != nil {
log.Error("Failed to get ENI limit")
return err
}
enisMax.Set(float64(c.maxENI))

c.maxENI = nodeMaxENI
c.unmanagedENI = numUnmanaged
c.maxIPsPerENI, err = c.awsClient.GetENIipLimit()
if err != nil {
log.Error("Failed to get IPs per ENI limit")
return err
}
ipMax.Set(float64(c.maxIPsPerENI * c.maxENI))

enis, err := c.awsClient.GetAttachedENIs()
if err != nil {
log.Error("Failed to retrieve ENI info")
return errors.New("ipamd init: failed to retrieve attached ENIs info")
}
c.updateIPStats(numUnmanaged)

_, vpcCIDR, err := net.ParseCIDR(c.awsClient.GetVPCIPv4CIDR())
if err != nil {
Expand Down Expand Up @@ -392,6 +397,11 @@ func (c *IPAMContext) nodeInit() error {
return err
}

func (c *IPAMContext) updateIPStats(unmanaged int) {
ipMax.Set(float64(c.maxIPsPerENI * (c.maxENI - unmanaged)))
enisMax.Set(float64(c.maxENI - unmanaged))
}

func (c *IPAMContext) getLocalPodsWithRetry() ([]*k8sapi.K8SPodInfo, error) {
var pods []*k8sapi.K8SPodInfo
var err error
Expand Down Expand Up @@ -628,12 +638,12 @@ func (c *IPAMContext) increaseIPPool() {
c.updateLastNodeIPPoolAction()
} else {
// If we did not add an IP, try to add an ENI instead.
if c.dataStore.GetENIs() < c.maxENI {
if c.dataStore.GetENIs() < (c.maxENI - c.unmanagedENI) {
if err = c.tryAllocateENI(); err == nil {
c.updateLastNodeIPPoolAction()
}
} else {
log.Debugf("Skipping ENI allocation as the instance's max ENI limit of %d is already reached", c.maxENI)
log.Debugf("Skipping ENI allocation as the instance's max ENI limit of %d is already reached (accounting for %d unmanaged ENIs)", c.maxENI, c.unmanagedENI)
}
}
}
Expand Down Expand Up @@ -786,7 +796,7 @@ func (c *IPAMContext) addENIaddressesToDataStore(ec2Addrs []*ec2.NetworkInterfac

// returns all addresses on ENI, the primary address on ENI, error
func (c *IPAMContext) getENIaddresses(eni string) ([]*ec2.NetworkInterfacePrivateIpAddress, string, error) {
ec2Addrs, _, err := c.awsClient.DescribeENI(eni)
ec2Addrs, _, _, err := c.awsClient.DescribeENI(eni)
if err != nil {
return nil, "", errors.Wrapf(err, "failed to find ENI addresses for ENI %s", eni)
}
Expand Down Expand Up @@ -941,13 +951,15 @@ func (c *IPAMContext) nodeIPPoolReconcile(interval time.Duration) {
}

log.Debug("Reconciling ENI/IP pool info...")
attachedENIs, err := c.awsClient.GetAttachedENIs()

allENIs, err := c.awsClient.GetAttachedENIs()
if err != nil {
log.Errorf("IP pool reconcile: Failed to get attached ENI info: %v", err.Error())
ipamdErrInc("reconcileFailedGetENIs")
return
}
attachedENIs, numUnmanaged := filterUnmanagedENIs(allENIs)
c.updateIPStats(numUnmanaged)
c.unmanagedENI = numUnmanaged

curENIs := c.dataStore.GetENIInfos()

Expand Down Expand Up @@ -1106,6 +1118,20 @@ func getMinimumIPTarget() int {
return noMinimumIPTarget
}

func filterUnmanagedENIs(enis []awsutils.ENIMetadata) ([]awsutils.ENIMetadata, int) {
numFiltered := 0
ret := make([]awsutils.ENIMetadata, 0, len(enis))
for _, eni := range enis {
if eni.Tags[eniNoManageTagKey] == "true" {
log.Debugf("skipping ENI %s: tagged with %s", eni.ENIID, eniNoManageTagKey)
numFiltered++
continue
}
ret = append(ret, eni)
}
return ret, numFiltered
}

// ipTargetState determines the number of IPs `short` or `over` our WARM_IP_TARGET,
// accounting for the MINIMUM_IP_TARGET
func (c *IPAMContext) ipTargetState() (short int, over int, enabled bool) {
Expand Down
16 changes: 9 additions & 7 deletions ipamd/ipamd_test.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,7 @@ func TestNodeInit(t *testing.T) {
{
PrivateIpAddress: &testAddr2, Primary: &notPrimary}}
mockAWS.EXPECT().GetPrimaryENI().Return(primaryENIid)
mockAWS.EXPECT().DescribeENI(primaryENIid).Return(eniResp, &attachmentID, nil)
mockAWS.EXPECT().DescribeENI(primaryENIid).Return(eniResp, map[string]string{}, &attachmentID, nil)

//secENIid
mockAWS.EXPECT().GetPrimaryENI().Return(primaryENIid)
Expand All @@ -140,7 +140,7 @@ func TestNodeInit(t *testing.T) {
{
PrivateIpAddress: &testAddr12, Primary: &notPrimary}}
mockAWS.EXPECT().GetPrimaryENI().Return(primaryENIid)
mockAWS.EXPECT().DescribeENI(secENIid).Return(eniResp, &attachmentID, nil)
mockAWS.EXPECT().DescribeENI(secENIid).Return(eniResp, map[string]string{}, &attachmentID, nil)
mockNetwork.EXPECT().SetupENINetwork(gomock.Any(), secMAC, secDevice, secSubnet)

mockAWS.EXPECT().GetLocalIPv4().Return(ipaddr01)
Expand All @@ -161,7 +161,7 @@ func TestNodeInit(t *testing.T) {
mockNetwork.EXPECT().UpdateRuleListBySrc(gomock.Any(), gomock.Any(), gomock.Any(), true)
// Add IPs
mockAWS.EXPECT().AllocIPAddresses(gomock.Any(), gomock.Any())
mockAWS.EXPECT().DescribeENI(gomock.Any()).Return(eniResp, &attachmentID, nil)
mockAWS.EXPECT().DescribeENI(gomock.Any()).Return(eniResp, map[string]string{}, &attachmentID, nil)

err := mockContext.nodeInit()
assert.NoError(t, err)
Expand Down Expand Up @@ -248,7 +248,8 @@ func testIncreaseIPPool(t *testing.T, useENIConfig bool) {
{PrivateIpAddress: &testAddr12, Primary: &notPrimary},
{PrivateIpAddress: &testAddr12, Primary: &notPrimary},
},
&attachmentID, nil)
map[string]string{}, &attachmentID, nil,
)

mockContext.increaseIPPool()
}
Expand Down Expand Up @@ -313,8 +314,7 @@ func TestTryAddIPToENI(t *testing.T) {
{PrivateIpAddress: &testAddr11, Primary: &primary},
{PrivateIpAddress: &testAddr12, Primary: &notPrimary},
{PrivateIpAddress: &testAddr12, Primary: &notPrimary},
},
&attachmentID, nil)
}, map[string]string{}, &attachmentID, nil)

mockContext.increaseIPPool()
}
Expand Down Expand Up @@ -356,7 +356,9 @@ func TestNodeIPPoolReconcile(t *testing.T) {
{
PrivateIpAddress: &testAddr1, Primary: &primary},
{
PrivateIpAddress: &testAddr2, Primary: &notPrimary}}, &attachmentID, nil)
PrivateIpAddress: &testAddr2, Primary: &notPrimary,
},
}, map[string]string{}, &attachmentID, nil)
mockAWS.EXPECT().GetPrimaryENI().Return(primaryENIid)

mockContext.nodeIPPoolReconcile(0)
Expand Down
38 changes: 28 additions & 10 deletions pkg/awsutils/awsutils.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -111,8 +111,8 @@ type APIs interface {
// GetAttachedENIs retrieves eni information from instance metadata service
GetAttachedENIs() (eniList []ENIMetadata, err error)

// DescribeENI returns the IPv4 addresses of ENI interface and ENI attachment ID
DescribeENI(eniID string) (addrList []*ec2.NetworkInterfacePrivateIpAddress, attachemdID *string, err error)
// DescribeENI returns the IPv4 addresses of ENI interface, tags, and the ENI attachment ID
DescribeENI(eniID string) (addrList []*ec2.NetworkInterfacePrivateIpAddress, tags map[string]string, attachemdID *string, err error)

// AllocIPAddress allocates an IP address for an ENI
AllocIPAddress(eniID string) error
Expand Down Expand Up @@ -185,6 +185,9 @@ type ENIMetadata struct {

// The ip addresses allocated for the network interface
LocalIPv4s []string

// Tags are the tags associated with this ENI in AWS
Tags map[string]string
}

// msSince returns milliseconds since start.
Expand Down Expand Up @@ -443,12 +446,18 @@ func (cache *EC2InstanceMetadataCache) getENIMetadata(macStr string) (ENIMetadat
if err != nil {
return ENIMetadata{}, errors.Wrapf(err, "get ENI metadata: failed to retrieve IPs and CIDR for ENI: %s", eniMAC)
}
_, tags, _, err := cache.DescribeENI(eni)
if err != nil {
return ENIMetadata{}, errors.Wrapf(err, "get ENI metadata: failed to describe ENI: %s, %v", eniMAC, err)
}
return ENIMetadata{
ENIID: eni,
MAC: eniMAC,
DeviceNumber: deviceNum,
SubnetIPv4CIDR: cidr,
LocalIPv4s: localIPv4s}, nil
LocalIPv4s: localIPv4s,
Tags: tags,
}, nil
}

// getIPsAndCIDR return list of IPs, CIDR, error
Expand Down Expand Up @@ -777,7 +786,7 @@ func (cache *EC2InstanceMetadataCache) freeENI(eniName string, maxBackoffDelay t
log.Infof("Trying to free ENI: %s", eniName)

// Find out attachment
_, attachID, err := cache.DescribeENI(eniName)
_, _, attachID, err := cache.DescribeENI(eniName)
if err != nil {
if err == ErrENINotFound {
log.Infof("ENI %s not found. It seems to be already freed", eniName)
Expand Down Expand Up @@ -852,9 +861,9 @@ func (cache *EC2InstanceMetadataCache) deleteENI(eniName string, maxBackoffDelay
return err
}

// DescribeENI returns the IPv4 addresses of interface and the attachment id
// return: private IP address, attachment id, error
func (cache *EC2InstanceMetadataCache) DescribeENI(eniID string) ([]*ec2.NetworkInterfacePrivateIpAddress, *string, error) {
// DescribeENI returns the IPv4 addresses, tags, and attachment id of the given ENI
// return: private IP address, tags, attachment id, error
func (cache *EC2InstanceMetadataCache) DescribeENI(eniID string) ([]*ec2.NetworkInterfacePrivateIpAddress, map[string]string, *string, error) {
eniIds := make([]*string, 0)
eniIds = append(eniIds, aws.String(eniID))
input := &ec2.DescribeNetworkInterfacesInput{NetworkInterfaceIds: eniIds}
Expand All @@ -865,14 +874,23 @@ func (cache *EC2InstanceMetadataCache) DescribeENI(eniID string) ([]*ec2.Network
if err != nil {
if aerr, ok := err.(awserr.Error); ok {
if aerr.Code() == "InvalidNetworkInterfaceID.NotFound" {
return nil, nil, ErrENINotFound
return nil, nil, nil, ErrENINotFound
}
}
awsAPIErrInc("DescribeNetworkInterfaces", err)
log.Errorf("Failed to get ENI %s information from EC2 control plane %v", eniID, err)
return nil, nil, errors.Wrap(err, "failed to describe network interface")
return nil, nil, nil, errors.Wrap(err, "failed to describe network interface")
}
return result.NetworkInterfaces[0].PrivateIpAddresses, result.NetworkInterfaces[0].Attachment.AttachmentId, nil
tags := make(map[string]string, len(result.NetworkInterfaces[0].TagSet))
for _, tag := range result.NetworkInterfaces[0].TagSet {
if tag.Key == nil || tag.Value == nil {
log.Errorf("nil tag on ENI: %v", eniID)
continue
}
tags[*tag.Key] = *tag.Value
}

return result.NetworkInterfaces[0].PrivateIpAddresses, tags, result.NetworkInterfaces[0].Attachment.AttachmentId, nil
}

// AllocIPAddress allocates an IP address for an ENI
Expand Down
Loading

0 comments on commit 57eab64

Please sign in to comment.