ibmcloud: slow VM startup

[squarti]

Describe the bug

The process-user-data service in peer pod VM makes requests to the VM metadata service to determine the VM flavor (AWS, Azure, etc) (see code). These requests are timing out in ibmcloud VMs (see logs below). Each request adds 30 seconds to the start up time.

root@podvm-nginx-794753b9:~# journalctl -u process-user-data
-- Logs begin at Wed 2024-08-28 17:43:09 UTC, end at Wed 2024-08-28 18:36:43 UTC. --
Aug 28 17:43:16 podvm-nginx-794753b9 systemd[1]: Starting Process user data...
Aug 28 17:43:47 podvm-nginx-794753b9 process-user-data[929]: failed to send request: Get "http://169.254.169.254/metadata/instance/compute?api-version=2021-01-01": dial tcp 169.254.169.254:80: i/o>
Aug 28 17:44:17 podvm-nginx-794753b9 process-user-data[929]: failed to send request: Get "http://169.254.169.254/latest/dynamic/instance-identity/document": dial tcp 169.254.169.254:80: i/o timeout
Aug 28 17:44:17 podvm-nginx-794753b9 process-user-data[929]: failed to send request: Get "http://metadata.google.internal/computeMetadata/v1/instance": dial tcp: lookup metadata.google.internal on>
Aug 28 17:44:17 podvm-nginx-794753b9 process-user-data[929]: 2024/08/28 17:44:17 [userdata/provision] unsupported user data provider, we extract and calculate initdata hash only.
Aug 28 17:44:17 podvm-nginx-794753b9 process-user-data[929]: 2024/08/28 17:44:17 [userdata/provision] File /run/peerpod/initdata not found, skipped initdata processing.
Aug 28 17:44:17 podvm-nginx-794753b9 systemd[1]: Finished Process user data.
root@podvm-nginx-794753b9:~# journalctl -u agent-protocol-forwarder
-- Logs begin at Wed 2024-08-28 17:43:09 UTC, end at Wed 2024-08-28 18:37:49 UTC. --
Aug 28 17:44:17 podvm-nginx-794753b9 systemd[1]: Starting Agent Protocol Forwarder...
Aug 28 17:44:18 podvm-nginx-794753b9 agent-protocol-forwarder[1185]: agent-protocol-forwarder version unknown
Aug 28 17:44:18 podvm-nginx-794753b9 agent-protocol-forwarder[1185]:   commit: 76e56780b519afbbf5944c72622fd1289103fc31-dirty
Aug 28 17:44:18 podvm-nginx-794753b9 agent-protocol-forwarder[1185]:   go: go1.21.12
Aug 28 17:44:18 podvm-nginx-794753b9 agent-protocol-forwarder[1185]: 2024/08/28 17:44:18 [forwarder] Starting agent-protocol-forwarder listener on address 0.0.0.0:15150
Aug 28 17:44:18 podvm-nginx-794753b9 agent-protocol-forwarder[1185]: 2024/08/28 17:44:18 [forwarder] TLS is configured. Configure TLS listener
Aug 28 17:44:18 podvm-nginx-794753b9 systemd[1]: Started Agent Protocol Forwarder.

How to reproduce

Run demo

CoCo version information

main

What TEE are you seeing the problem on

None

Failing command and relevant log output

No response