Enhance LinuxCollector to support detecting multiple app VIF IPs

docs/APP-CONNECTIVITY.md

@@ -216,6 +216,25 @@ to retrieve cloud-init configuration, obtain information from EVE (e.g. device U
 hostname, external IP address) or to download [patch envelopes](PATCH-ENVELOPES.md).
 More information about metadata server can be found in [ECO-METADATA.md](ECO-METADATA.md).
 
+#### IPAM
+
+Every Local Network Instance must be configured with an IPv4 network subnet and an IP
+range within this subnet for automatic IP allocations. Host IP addresses from this subnet
+that do not fall within the IP range are available for manual assignment.
+
+Whether an IP address is selected manually or dynamically assigned by EVE from the configured
+IP range, an internal DHCP server is used to distribute these IP addresses to applications.
+Container applications are deployed inside a "shim VM", which EVE prepares, ensuring that
+a DHCP client is running for every virtual interface connected to a network instance
+This guarantees that the IP address is received and applied before the application starts.
+In contrast, VM applications are responsible for starting their own DHCP client and applying
+the received IP addresses.
+
+Regardless of the application type, EVE does not automatically assume that the allocated
+IP address is actually in use. Instead, it monitors the set of IP leases granted by the internal
+DHCP server and updates the set of application IP addresses in the published info messages
+accordingly.
+
 ### Switch Network Instance
 
 Switch Network Instance is a simple L2-only bridge between connected applications and
@@ -241,6 +260,52 @@ inbound ACL rules are that much more important.
 A metadata HTTP server is run for a switch network instance only if it has a port attached
 that has an IP address.
 
+#### IP address detection
+
+Unlike a Local Network Instance, a switch network instance is configured without any IP
+configuration, and EVE does not run an internal DHCP server. Instead, if IP connectivity
+is required, IP addresses must be assigned statically within the connected applications
+or provided by an external DHCP server or another application offering DHCP services.
+
+Since EVE is not in control of IP address allocations and leases, it must monitor application
+traffic to learn which IP addresses are being used and report this information to the controller.
+
+In the case of an external DHCP server (IPv4), EVE captures the DHCPACK packet from the server,
+which confirms the leased IP address. Because EVE manages MAC address allocations, it knows
+the MAC address of every application's virtual interface (VIF). It can then map the CHADDR
+(Client Hardware Address) attribute to the corresponding application VIF and learn the assigned
+IP address from the YIADDR (your, i.e. client, IP Address) attribute. Additionally, EVE reads
+the DHCP option 51 (Lease Time), if available, to determine how long the leased IP address
+is valid. If EVE does not observe an IP renewal within this period, it assumes that the IP address
+is no longer in use and reports this change to the controller.
+
+For statically assigned IPv4 addresses, EVE captures both ARP reply and request packets to learn
+the application VIF IP assignment from either Sender IP + MAC or Target IP + MAC attribute
+pairs. Since ARP cache entries have a limited lifetime — typically around 2 minutes — EVE expects
+to see at least one ARP packet for every assigned IP within a 10-minute window (this is not
+configurable). If no ARP packet is observed within this period for a previously detected IP
+assignment, EVE assumes that the IP address has been removed and reports this change to
+the controller. EVE also captures ARP packets for IP addresses configured via DHCP, but these
+are ignored as the information from the previously captured DHCPACK takes precedence.
+Note that ARP-based IP detection is enabled by default but can be disabled by setting
+the configuration item `network.switch.enable.arpsnoop` to `false`. Change in this config
+options will apply to already deployed switch network instances.
+
+For an external DHCPv6 server, EVE captures DHCPv6 REPLY messages. It learns the target MAC
+address from the DUID option (Client Identifier, option code 1), while the IPv6 address
+and its valid lifetime are provided by the IA Address (option code 5).
+
+To learn IPv6 addresses assigned using SLAAC (Stateless Address Auto Configuration),
+EVE captures unicast ICMPv6 Neighbor Solicitation messages. These are sent from the interface
+with the assigned IPv6 address to check if the address is free or already in use by another
+host — a process known as Duplicate Address Detection (DAD). The ICMPv6 packet sent to detect
+IP duplicates for a particular VIF IP will have the VIF MAC address as the source address
+in the Ethernet header. EVE uses this, along with the "Target Address" field from the ICMPv6
+header, to identify the assigned IPv6 address.
+
+EVE is capable of detecting multiple IPs assigned to the same VIF MAC address. This is commonly
+seen when applications use VLAN sub-interfaces, which share the parent interface's MAC address.
+
 ### Network Instance Ports
 
 Network instances can be configured with one or more network adapters, which will be used

docs/CONFIG-PROPERTIES.md

@@ -55,7 +55,7 @@
 | netdump.topic.maxcount | integer | 10 | maximum number of netdumps that can be published for each topic. The oldest netdump is unpublished should a new netdump exceed the limit.
 | netdump.downloader.with.pcap | boolean | false | include packet captures inside netdumps for download requests. However, even if enabled, TCP segments carrying non-empty payload (i.e. content which is being downloaded) are excluded and the overall PCAP size is limited to 64MB. |
 | netdump.downloader.http.with.fieldvalue | boolean | false | include HTTP header field values in captured network traces for download requests (beware: may contain secrets, such as datastore credentials). |
-| network.switch.enable.arpsnoop | boolean | true | enable ARP Snooping on switch Network Instance, may need a device reboot to take effect |
+| network.switch.enable.arpsnoop | boolean | true | enable ARP Snooping on switch Network Instances |
 | wwan.query.visible.providers | bool | false | enable to periodically (once per hour) query the set of visible cellular service providers and publish them under WirelessStatus (for every modem) |
 | network.local.legacy.mac.address | bool | false | enables legacy MAC address generation for local network instances for those EVE nodes where changing MAC addresses in applications will lead to incorrect network configuration |
 

pkg/pillar/cmd/msrv/msrv_test.go

@@ -47,7 +47,14 @@ func TestPostKubeconfig(t *testing.T) {
 		},
 		AppNetAdapterList: []types.AppNetAdapterStatus{
 			{
-				AllocatedIPv4Addr: net.ParseIP("192.168.1.1"),
+				AssignedAddresses: types.AssignedAddrs{
+					IPv4Addrs: []types.AssignedAddr{
+						{
+							Address: net.ParseIP("192.168.1.1"),
+						},
+					},
+					IPv6Addrs: nil,
+				},
 			},
 		},
 	})
@@ -83,7 +90,11 @@ func TestPostKubeconfig(t *testing.T) {
 	niStatus := types.NetworkInstanceStatus{
 		NetworkInstanceInfo: types.NetworkInstanceInfo{
 			IPAssignments: map[string]types.AssignedAddrs{"k": {
-				IPv4Addr: net.ParseIP("192.168.1.1"),
+				IPv4Addrs: []types.AssignedAddr{
+					{
+						Address: net.ParseIP("192.168.1.1"),
+					},
+				},
 			}},
 		},
 	}
@@ -161,7 +172,14 @@ func TestRequestPatchEnvelopes(t *testing.T) {
 		},
 		AppNetAdapterList: []types.AppNetAdapterStatus{
 			{
-				AllocatedIPv4Addr: net.ParseIP("192.168.1.1"),
+				AssignedAddresses: types.AssignedAddrs{
+					IPv4Addrs: []types.AssignedAddr{
+						{
+							Address: net.ParseIP("192.168.1.1"),
+						},
+					},
+					IPv6Addrs: nil,
+				},
 			},
 		},
 	})
@@ -309,7 +327,14 @@ func TestHandleAppInstanceDiscovery(t *testing.T) {
 		},
 		AppNetAdapters: []types.AppNetAdapterStatus{
 			{
-				AllocatedIPv4Addr: net.ParseIP("192.168.1.1"),
+				AssignedAddresses: types.AssignedAddrs{
+					IPv4Addrs: []types.AssignedAddr{
+						{
+							Address: net.ParseIP("192.168.1.1"),
+						},
+					},
+					IPv6Addrs: nil,
+				},
 				AppNetAdapterConfig: types.AppNetAdapterConfig{
 					IfIdx:           2,
 					AllowToDiscover: true,
@@ -320,8 +345,15 @@ func TestHandleAppInstanceDiscovery(t *testing.T) {
 	err = appInstanceStatus.Publish(u.String(), a)
 	g.Expect(err).ToNot(gomega.HaveOccurred())
 	discoverableNet := types.AppNetAdapterStatus{
-		AllocatedIPv4Addr: net.ParseIP("192.168.1.2"),
-		VifInfo:           types.VifInfo{VifConfig: types.VifConfig{Vif: "eth0"}},
+		AssignedAddresses: types.AssignedAddrs{
+			IPv4Addrs: []types.AssignedAddr{
+				{
+					Address: net.ParseIP("192.168.1.2"),
+				},
+			},
+			IPv6Addrs: nil,
+		},
+		VifInfo: types.VifInfo{VifConfig: types.VifConfig{Vif: "eth0"}},
 	}
 	a1 := types.AppInstanceStatus{
 		UUIDandVersion: types.UUIDandVersion{
@@ -367,7 +399,7 @@ func TestHandleAppInstanceDiscovery(t *testing.T) {
 	expected := map[string][]msrv.AppInstDiscovery{
 		u1.String(): {{
 			Port:    discoverableNet.Vif,
-			Address: discoverableNet.AllocatedIPv4Addr.String(),
+			Address: discoverableNet.AssignedAddresses.IPv4Addrs[0].Address.String(),
 		}},
 	}
 	g.Expect(got).To(gomega.BeEquivalentTo(expected))

pkg/pillar/cmd/msrv/pubsub.go

@@ -24,14 +24,42 @@ func (srv *Msrv) lookupAppNetworkStatusByAppIP(ip net.IP) *types.AppNetworkStatu
 	for _, st := range items {
 		status := st.(types.AppNetworkStatus)
 		for _, adapterStatus := range status.AppNetAdapterList {
-			if adapterStatus.AllocatedIPv4Addr.Equal(ip) {
-				return &status
+			for _, adapterIP := range adapterStatus.AssignedAddresses.IPv4Addrs {
+				if adapterIP.Address.Equal(ip) {
+					return &status
+				}
+			}
+			for _, adapterIP := range adapterStatus.AssignedAddresses.IPv6Addrs {
+				if adapterIP.Address.Equal(ip) {
+					return &status
+				}
 			}
 		}
 	}
 	return nil
 }
 
+func (srv *Msrv) lookupAppInstStatusByAppIP(ip net.IP) (*types.AppInstanceStatus, bool) {
+	sub := srv.subAppInstanceStatus
+	items := sub.GetAll()
+	for _, sc := range items {
+		status := sc.(types.AppInstanceStatus)
+		for _, adapterStatus := range status.AppNetAdapters {
+			for _, adapterIP := range adapterStatus.AssignedAddresses.IPv4Addrs {
+				if adapterIP.Address.Equal(ip) {
+					return &status, true
+				}
+			}
+			for _, adapterIP := range adapterStatus.AssignedAddresses.IPv6Addrs {
+				if adapterIP.Address.Equal(ip) {
+					return &status, true
+				}
+			}
+		}
+	}
+	return nil, false
+}
+
 func (srv *Msrv) getExternalIPsForApp(remoteIP net.IP) ([]net.IP, int) {
 	netstatus := srv.lookupNetworkInstanceStatusByAppIP(remoteIP)
 	if netstatus == nil {
@@ -80,11 +108,13 @@ func (srv *Msrv) lookupNetworkInstanceStatusByAppIP(
 	for _, st := range items {
 		status := st.(types.NetworkInstanceStatus)
 		for _, addrs := range status.IPAssignments {
-			if ip.Equal(addrs.IPv4Addr) {
-				return &status
+			for _, assignedIP := range addrs.IPv4Addrs {
+				if ip.Equal(assignedIP.Address) {
+					return &status
+				}
 			}
-			for _, nip := range addrs.IPv6Addrs {
-				if ip.Equal(nip) {
+			for _, assignedIP := range addrs.IPv6Addrs {
+				if ip.Equal(assignedIP.Address) {
 					return &status
 				}
 			}
@@ -483,21 +513,6 @@ func (srv *Msrv) handleAppInstDelete(ctxArg interface{}, key string,
 	srv.Log.Functionf("handleAppInstDelete(%s) done", key)
 }
 
-func (srv *Msrv) lookupAppInstStatusByAppIP(ip net.IP) (*types.AppInstanceStatus, bool) {
-	sub := srv.subAppInstanceStatus
-	items := sub.GetAll()
-	for _, sc := range items {
-		status := sc.(types.AppInstanceStatus)
-		for _, adapterStatus := range status.AppNetAdapters {
-			if adapterStatus.AllocatedIPv4Addr.Equal(ip) {
-				return &status, adapterStatus.AllowToDiscover
-			}
-		}
-	}
-
-	return nil, false
-}
-
 // AppInstDiscovery is a struct which AppInstances see, when they request discoverable
 type AppInstDiscovery struct {
 	Port    string `json:"port"`
@@ -514,13 +529,18 @@ func (srv *Msrv) composeAppInstancesIPAddresses(UUIDToSkip uuid.UUID) map[string
 		}
 		adapters := make([]AppInstDiscovery, 0)
 		for _, adapterStatus := range status.AppNetAdapters {
-			if adapterStatus.AllocatedIPv4Addr == nil || adapterStatus.AllocatedIPv4Addr.String() == "" {
-				continue
+			for _, ip := range adapterStatus.AssignedAddresses.IPv4Addrs {
+				adapters = append(adapters, AppInstDiscovery{
+					Port:    adapterStatus.Vif,
+					Address: ip.Address.String(),
+				})
+			}
+			for _, ip := range adapterStatus.AssignedAddresses.IPv6Addrs {
+				adapters = append(adapters, AppInstDiscovery{
+					Port:    adapterStatus.Vif,
+					Address: ip.Address.String(),
+				})
 			}
-			adapters = append(adapters, AppInstDiscovery{
-				Port:    adapterStatus.Vif,
-				Address: adapterStatus.AllocatedIPv4Addr.String(),
-			})
 		}
 		res[status.UUIDandVersion.UUID.String()] = adapters
 	}

pkg/pillar/cmd/zedagent/handleconfig.go

@@ -1210,8 +1210,22 @@ func updateLocalServerMap(getconfigCtx *getconfigContext, localServerURL string)
 				continue
 			}
 			if localServerIP != nil {
-				// check if the defined IP of localServer equals the allocated IP of the app
-				if adapterStatus.AllocatedIPv4Addr.Equal(localServerIP) {
+				// Check if the defined IP of localServer equals one of the IPs
+				// allocated to the app.
+				var matchesApp bool
+				for _, ip := range adapterStatus.AssignedAddresses.IPv4Addrs {
+					if ip.Address.Equal(localServerIP) {
+						matchesApp = true
+						break
+					}
+				}
+				for _, ip := range adapterStatus.AssignedAddresses.IPv6Addrs {
+					if ip.Address.Equal(localServerIP) {
+						matchesApp = true
+						break
+					}
+				}
+				if matchesApp {
 					srvAddr := localServerAddr{
 						localServerAddr: localServerURL,
 						bridgeIP:        adapterStatus.BridgeIPAddr,

pkg/pillar/cmd/zedagent/handlemetrics.go

@@ -1112,16 +1112,18 @@ func PublishAppInfoToZedCloud(ctx *zedagentContext, uuid string,
 		for _, ifname := range ifNames {
 			networkInfo := new(info.ZInfoNetwork)
 			networkInfo.LocalName = *proto.String(ifname)
-			ipv4Addr, ipv6Addrs, allocated, macAddr, ipAddrMismatch :=
-				getAppIP(ctx, aiStatus, ifname)
-			if ipv4Addr != nil {
-				networkInfo.IPAddrs = append(networkInfo.IPAddrs, ipv4Addr.String())
+			addrs, hasIPv4Addr, macAddr, ipAddrMismatch :=
+				getAppIPs(ctx, aiStatus, ifname)
+			for _, ipv4Addr := range addrs.IPv4Addrs {
+				networkInfo.IPAddrs = append(networkInfo.IPAddrs,
+					ipv4Addr.Address.String())
 			}
-			for _, ipv6Addr := range ipv6Addrs {
-				networkInfo.IPAddrs = append(networkInfo.IPAddrs, ipv6Addr.String())
+			for _, ipv6Addr := range addrs.IPv6Addrs {
+				networkInfo.IPAddrs = append(networkInfo.IPAddrs,
+					ipv6Addr.Address.String())
 			}
 			networkInfo.MacAddr = *proto.String(macAddr.String())
-			networkInfo.Ipv4Up = allocated
+			networkInfo.Ipv4Up = hasIPv4Addr
 			networkInfo.IpAddrMisMatch = ipAddrMismatch
 			name := appIfnameToName(aiStatus, ifname)
 			log.Tracef("app %s/%s localName %s devName %s",
@@ -1560,21 +1562,22 @@ func sendMetricsProtobuf(ctx *getconfigContext,
 
 // Use the ifname/vifname to find the AppNetAdapter status
 // and from there the (ip, allocated, mac) addresses for the app
-func getAppIP(ctx *zedagentContext, aiStatus *types.AppInstanceStatus,
-	vifname string) (net.IP, []net.IP, bool, net.HardwareAddr, bool) {
+func getAppIPs(ctx *zedagentContext, aiStatus *types.AppInstanceStatus,
+	vifname string) (types.AssignedAddrs, bool, net.HardwareAddr, bool) {
 
 	log.Tracef("getAppIP(%s, %s)", aiStatus.Key(), vifname)
 	for _, adapterStatus := range aiStatus.AppNetAdapters {
 		if adapterStatus.VifUsed != vifname {
 			continue
 		}
-		log.Tracef("getAppIP(%s, %s) found AppIP v4: %s, v6: %s, ipv4 assigned %v mac %s",
-			aiStatus.Key(), vifname, adapterStatus.AllocatedIPv4Addr,
-			adapterStatus.AllocatedIPv6List, adapterStatus.IPv4Assigned, adapterStatus.Mac)
-		return adapterStatus.AllocatedIPv4Addr, adapterStatus.AllocatedIPv6List, adapterStatus.IPv4Assigned,
+		log.Tracef("getAppIP(%s, %s) found AppIPs v4: %v, v6: %v, ipv4 assigned %v mac %s",
+			aiStatus.Key(), vifname, adapterStatus.AssignedAddresses.IPv4Addrs,
+			adapterStatus.AssignedAddresses.IPv6Addrs, adapterStatus.IPv4Assigned,
+			adapterStatus.Mac)
+		return adapterStatus.AssignedAddresses, adapterStatus.IPv4Assigned,
 			adapterStatus.Mac, adapterStatus.IPAddrMisMatch
 	}
-	return nil, nil, false, nil, false
+	return types.AssignedAddrs{}, false, nil, false
 }
 
 func createVolumeInstanceMetrics(ctx *zedagentContext, reportMetrics *metrics.ZMetricMsg) {

pkg/pillar/cmd/zedagent/handlenetworkinstance.go

@@ -8,7 +8,6 @@ package zedagent
 import (
 	"bytes"
 	"fmt"
-	"net"
 	"time"
 
 	"github.com/golang/protobuf/ptypes/timestamp"
@@ -134,11 +133,13 @@ func prepareAndPublishNetworkInstanceInfoMsg(ctx *zedagentContext,
 		for mac, addrs := range status.IPAssignments {
 			assignment := new(zinfo.ZmetIPAssignmentEntry)
 			assignment.MacAddress = mac
-			if !addrs.IPv4Addr.Equal(net.IP{}) {
-				assignment.IpAddress = append(assignment.IpAddress, addrs.IPv4Addr.String())
+			for _, assignedIP := range addrs.IPv4Addrs {
+				assignment.IpAddress = append(assignment.IpAddress,
+					assignedIP.Address.String())
 			}
-			for _, ip := range addrs.IPv6Addrs {
-				assignment.IpAddress = append(assignment.IpAddress, ip.String())
+			for _, assignedIP := range addrs.IPv6Addrs {
+				assignment.IpAddress = append(assignment.IpAddress,
+					assignedIP.Address.String())
 			}
 			info.IpAssignments = append(info.IpAssignments, assignment)
 		}