ec2, hotplug: race between udev add event and IMDS data not being available

[aciba90]

Bug report

Cloud-init hits what @nmeyerhans mentioned in #4799 (comment):

One thing to be aware of when accessing IMDS in response to udev add events is that IMDS propagates asynchronously, and the data corresponding with a newly attached network interface may not be complete at the time that the handler is running.

Sometimes, the IMDS does not have the full metadata about a hotplugged NIC when triggered by a hotplug udev event.

We can see the difference in instance network metadata after a reboot:

--- tmp/pre/cloud-init-logs-2024-06-05/run/cloud-init/instance-data.json        2024-06-05 11:46:07.000000000 +0200
+++ tmp/post/cloud-init-logs-2024-06-05/run/cloud-init/instance-data.json       2024-06-05 12:49:30.000000000 +0200
@@ -91,7 +91,7 @@
      "info": {
       "AccountId": "937157663530",
       "Code": "Success",
-      "LastUpdated": "2024-06-05T09:44:21Z"
+      "LastUpdated": "2024-06-05T10:33:52Z"
      }
     }
    },
@@ -141,6 +141,7 @@
       "06:dc:23:8e:6c:bc": {
        "device-number": "1",
        "interface-id": "eni-08f52ea023734d621",
+       "ipv6s": "2a05:d011:311:a00:c4f1:eb1b:92d2:200e",
        "local-hostname": "ip-192-168-5-5.eu-south-2.compute.internal",
        "local-ipv4s": "192.168.5.5",
        "mac": "06:dc:23:8e:6c:bc",
@@ -155,8 +156,11 @@
        ],
        "subnet-id": "subnet-0877465f33a6d8dde",
        "subnet-ipv4-cidr-block": "192.168.0.0/20",
+       "subnet-ipv6-cidr-blocks": "2a05:d011:311:a00:0:0:0:0/64",
        "vpc-id": "vpc-031a740d79767510f",
-       "vpc-ipv4-cidr-block": "192.168.0.0/20"
+       "vpc-ipv4-cidr-block": "192.168.0.0/20",
+       "vpc-ipv4-cidr-blocks": "192.168.0.0/20",
+       "vpc-ipv6-cidr-blocks": "2a05:d011:311:a00:0:0:0:0/56"
       }
      }
     }

Steps to reproduce the problem

export CLOUD_INIT_CLOUD_INIT_SOURCE=cloud-init_all.deb
export CLOUD_INIT_PLATFORM=ec2
export CLOUD_INIT_OS_IMAGE=noble

while true; do
	tox -e integration-tests -- --pdb \
		tests/integration_tests/modules/test_hotplug.py::test_multi_nic_hotplug_vpc
done

And observe:

    @pytest.mark.skipif(CURRENT_RELEASE <= FOCAL, reason="See LP: #2055397")
    @pytest.mark.skipif(PLATFORM != "ec2", reason="test is ec2 specific")
    def test_multi_nic_hotplug_vpc(setup_image, session_cloud: IntegrationCloud):
        """Tests that additional secondary NICs are routable from local
        networks after the hotplug hook is executed when network updates
        are configured on the HOTPLUG event."""
        with session_cloud.launch(
            user_data=USER_DATA
        ) as client, session_cloud.launch() as bastion:
            ips_before = _get_ip_addr(client)
            primary_priv_ip4 = ips_before[1].ip4
            primary_priv_ip6 = ips_before[1].ip6
            client.instance.add_network_interface(ipv6_address_count=1)
    
            _wait_till_hotplug_complete(client)
            log_content = client.read_from_file("/var/log/cloud-init.log")
            verify_clean_log(log_content)
    
            netplan_cfg = client.read_from_file("/etc/netplan/50-cloud-init.yaml")
            config = yaml.safe_load(netplan_cfg)
    
            ips_after_add = _get_ip_addr(client)
            secondary_priv_ip4 = ips_after_add[2].ip4
            secondary_priv_ip6 = ips_after_add[2].ip6
            assert primary_priv_ip4 != secondary_priv_ip4
    
            new_addition = [
                ip for ip in ips_after_add if ip.ip4 == secondary_priv_ip4
            ][0]
            assert new_addition.interface in config["network"]["ethernets"]
            new_nic_cfg = config["network"]["ethernets"][new_addition.interface]
            assert "routing-policy" in new_nic_cfg
>           assert [
                {"from": secondary_priv_ip4, "table": 101},
                {"from": secondary_priv_ip6, "table": 101},
            ] == new_nic_cfg["routing-policy"]
E           AssertionError: assert [{'from': '192.168.3.64', 'table': 101}, {'from': 'fe80::ca:54ff:fee0:3c8f', 'table': 101}] == [{'from': '192.168.3.64', 'table': 101}]
E             
E             Left contains one more item: {'from': 'fe80::ca:54ff:fee0:3c8f', 'table': 101}
E             
E             Full diff:
E               [
E                   {
E                       'from': '192.168.3.64',
E                       'table': 101,
E                   },
E             +     {
E             +         'from': 'fe80::ca:54ff:fee0:3c8f',
E             +         'table': 101,
E             +     },
E               ]

Environment details

• Cloud-init version: 24.1 and tip of main.
• Operating System Distribution: Ubuntu
• Cloud provider, platform or installer type: Ec2

cloud-init logs

cloud-init.tar.gz
cloud-init-post-reboot.tar.gz

[aciba90]

Some kind of retry / waiting has to be added when required networking properties from the IMDS are missing.

[TheRealFalcon]

@aciba90 , so this has always been an issue with hotplug? We just happen to be seeing it now?

[aciba90]

@TheRealFalcon: I think so, the lack of #5271 was shadowing this race condition, which does not happen all times.

I believe #5283 is due to this race.

[aciba90]

Tracking in: SF#00387392.

[aciba90]

Test disabled in #5503.

[aciba90]

Per responses in SF#00387392, there is no better solution to synchronize with the IMDS, but implementing a wait / retry mechanism.

Another solution, outlined by Noah in [1], would be to migrate our current implementation and configure PBR as a dhcp exit hook. This would have access to all required information, but we would need to implement the hooks for every dhcp client that cloud-init supports, or at least minimally for every default dhcp in the current Ubuntu supported releases:

2. We can configure routing in a dhclient exit hook. This should have
   access to all the details that dhclient has configured on the interface
   via the environment. However, it tightens our coupling with dhclient,
   which may not be desirable. There's been some talk of moving to
   systemd-networkd for interface configuration.

In summary, I see two possible solutions:

1. Implement a wait / retry mechanism on top of the current implementation: udev rules triggering after a NIC is added and getting the info from the IMDS.
2. Implement dhcp hooks.

[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=963826

[TheRealFalcon]

Removing from the 24.3 milestone as this is likely going to be addressed in the 25.04 lifecycle

[aciba90]

Tracking in SC-1850.