No NIC present in isolated Windows container using WinNat on Windows 11

[riverar]

Describe the bug
After starting an isolated Hyper-V container on Windows 11 22H2+, the container does not have a vNIC attached and/or networking does not function at all.

This is easy to reproduce in a virtualized environment 100% of the time.

Does not reproduce with non-isolated containers.

To Reproduce

1. Set up a vanilla Windows install with Containers and Hyper-V features enabled
2. Download/extract containerd into C:\containerd
3. Download/extract windows-cni into C:\containerd\cni\bin
4. (As administrator) Issue command: New-Item -ItemType SymbolicLink -Path "C:\Program Files\containerd\" -Target "C:\containerd\"
5. Create the configuration files provided below, at the exact locations specified below
6. (As administrator) Start containerd.exe
7. (As administrator) Start container (./ctr run --cni --isolated -t --rm mcr.microsoft.com/windows:ltsc2019-amd64 test)
8. Observe ipconfig in container is blank / networking does not work

Expected behavior
Container to start with a vNIC attached and communicating with the network normally.

Configuration:

• Edition: Windows 11 22H2 22621.4317, Windows vNext 27774.rs_prerelease.250111-1221, Windows Server 2022 20348.3091
• Base Image being used: mcr.microsoft.com/windows:ltsc2019-amd64
• CNI: windows-cni 0.3.1 (release)
• Container engine: containerd 2.0.1 (release)
• Container engine version:

  Client:
    Version:  v2.0.1
    Revision: 88aa2f531d6c2922003cc7929e51daf1c14caa0a
    Go version: go1.23.4
  
  Server:
    Version:  v2.0.1
    Revision: 88aa2f531d6c2922003cc7929e51daf1c14caa0a
    UUID: c71c0b31-fa2e-44a5-b5e3-4cea273b5a18
  

Configuration files

C:\containerd\config.toml

version = 3
[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runhcs-wcow-process]
  cni_conf_dir = "C:\\containerd\\cni\\conf"

[plugins.'io.containerd.cri.v1.runtime'.cni]
  bin_dir = "C:\\containerd\\cni\\bin"
  conf_dir = "C:\\containerd\\cni\\conf"

C:\containerd\cni\conf\0-containerd-nat.conf

{
  "cniVersion": "0.3.0",
  "name": "nat",
  "type": "nat"
}

Additional context / logs

wincni.log

{"level":"debug","msg":"[cni-net] Plugin wcn-net version .","time":"2025-01-22T17:18:49-08:00"}
{"level":"debug","msg":"[net] Network interface: {Index:17 MTU:1500 Name:vEthernet (Default Switch) HardwareAddr:00:15:5d:01:de:cf Flags:up|broadcast|multicast|running} with IP addresses: [fe80::af64:80c4:39e7:77f1/64 172.22.208.1/20]","time":"2025-01-22T17:18:49-08:00"}
{"level":"debug","msg":"[net] Network interface: {Index:6 MTU:1500 Name:Ethernet (Kernel Debugger) HardwareAddr:16:89:d5:78:83:25 Flags:up|broadcast|multicast|running} with IP addresses: [fe80::ebe4:818:9a61:88ab/64 192.168.1.139/24]","time":"2025-01-22T17:18:49-08:00"}
{"level":"debug","msg":"[net] Network interface: {Index:1 MTU:-1 Name:Loopback Pseudo-Interface 1 HardwareAddr: Flags:up|loopback|multicast|running} with IP addresses: [::1/128 127.0.0.1/8]","time":"2025-01-22T17:18:49-08:00"}
{"level":"debug","msg":"[net] Network interface: {Index:32 MTU:1500 Name:vEthernet (nat) HardwareAddr:00:15:5d:cd:bc:43 Flags:up|broadcast|multicast|running} with IP addresses: [fe80::7d57:ba99:ad33:8d5a/64 172.27.144.1/20]","time":"2025-01-22T17:18:49-08:00"}
{"level":"debug","msg":"[cni-net] Plugin started.","time":"2025-01-22T17:18:49-08:00"}
{"level":"debug","msg":"[cni-net] Processing ADD command with args {ContainerID:default-test Netns:2c094b9e-af3a-4dc1-97d8-821b68603d23 IfName:eth0 Args: Path:C:\\Program Files\\containerd\\cni\\bin}.","time":"2025-01-22T17:18:49-08:00"}
{"level":"debug","msg":"[cni-net] Read network configuration \u0026{CniVersion:0.3.0 Name:nat Type:nat Ipam:{Type: Environment: AddrSpace: Subnet: Address: QueryInterval: Routes:[]} DNS:{Nameservers:[] Domain: Search:[] Options:[]} OptionalFlags:{LocalRoutePortMapping:false AllowAclPortMapping:false ForceBridgeGateway:false EnableDualStack:false LoopbackDSR:false GatewayFromAdditionalRoutes:false} RuntimeConfig:{PortMappings:[] DNS:{Servers:[] Searches:[] Options:[]}} AdditionalRoutes:[] AdditionalArgs:[]}.","time":"2025-01-22T17:18:49-08:00"}
{"level":"info","msg":"[cni-net] Dual stack is disabled","time":"2025-01-22T17:18:49-08:00"}
{"level":"debug","msg":"Parsing port mappings from []","time":"2025-01-22T17:18:49-08:00"}
{"level":"debug","msg":"[cni-net] Found network 76177bd4-a2b5-43a6-9cb8-a37d9d903123 with subnet [{{172.27.144.0 fffff000} 172.27.144.1 [{NetworkPolicy [123 34 84 121 112 101 34 58 34 86 76 65 78 34 44 34 83 101 116 116 105 110 103 115 34 58 123 34 73 115 111 108 97 116 105 111 110 73 100 34 58 49 125 125]}]}].","time":"2025-01-22T17:18:49-08:00"}
{"level":"debug","msg":"[cni-net] Creating a new Endpoint","time":"2025-01-22T17:18:49-08:00"}
{"level":"debug","msg":"hcn::HostComputeEndpoint::Create id=","time":"2025-01-22T17:18:49-08:00"}
{"level":"debug","msg":"hcn::HostComputeEndpoint::Create JSON: {\"Name\":\"default-test_nat\",\"HostComputeNetwork\":\"76177bd4-a2b5-43a6-9cb8-a37d9d903123\",\"Dns\":{},\"Routes\":[{\"DestinationPrefix\":\"0.0.0.0/0\"}],\"Health\":{\"Extra\":{}},\"SchemaVersion\":{\"Major\":2,\"Minor\":0}}","time":"2025-01-22T17:18:49-08:00"}
{"level":"debug","msg":"hcn::HostComputeEndpoint::AddNamespaceEndpoint id=5e6298c2-4638-494e-be6b-ff6fef942c06","time":"2025-01-22T17:18:49-08:00"}
{"level":"debug","msg":"hcn::HostComputeNamespace::ModifyNamespaceSettings id=2c094b9e-af3a-4dc1-97d8-821b68603d23","time":"2025-01-22T17:18:49-08:00"}
{"level":"debug","msg":"[cni-net] result: \u0026{CNIVersion:0.3.0 Interfaces:[{Name:eth0 Mac:\u0000\u0015];G Sandbox:}] IPs:[{Version:4 Interface:0xc000206b78 Address:{IP:172.27.148.101 Mask:fffff000} Gateway:172.27.144.1}] Routes:[] DNS:{Nameservers:[] Domain: Search:[] Options:[]}}","time":"2025-01-22T17:18:49-08:00"}
{"level":"debug","msg":"[cni-net] Plugin stopped.","time":"2025-01-22T17:18:49-08:00"}

hcsdiag list all

////////////////////////NETWORKS///////////////////////

Network : C08CB7B8-9B3C-408E-8E30-5E16A3AEB444
    Name             : Default Switch
    Type             : ICS
    Subnet Address   : 172.22.208.0/20
    Gateway          : 172.22.208.1

Network : 76177BD4-A2B5-43A6-9CB8-A37D9D903123
    Name             : nat
    Type             : NAT
    Subnet Address   : 172.27.144.0/20
    Gateway          : 172.27.144.1

///////////////////////ENDPOINTS///////////////////////

Endpoint         : 5e6298c2-4638-494e-be6b-ff6fef942c06
    Name             : default-test_nat
    IP Address       : 172.27.148.101

//////////////////////NAMESPACES///////////////////////

Namespace        : 77a05b01-0eff-4941-a6c0-c9005c516c7c

Namespace        : 910f7d92-ba2d-4c3f-98ae-7c0ac590d2dc

Namespace        : 1a66f6bd-8bc1-468b-9e5b-06fe6694a0b5

Namespace        : 2c094b9e-af3a-4dc1-97d8-821b68603d23
    Endpoint IDs     : 5E6298C2-4638-494E-BE6B-FF6FEF942C06

Namespace        : bac6ccf7-6ba8-4130-a8ee-3fffb9bac1e4

/////////////////////LOADBALANCERS/////////////////////

Get-HnsEndpoint | ? ID -like ""

PS C:\Users\Rafael> Get-HnsEndpoint | ? ID -like "5e6298c2-4638-494e-be6b-ff6fef942c06"

ID                 : 5e6298c2-4638-494e-be6b-ff6fef942c06
Name               : default-test_nat
Version            : 64424509440
AdditionalParams   :
Resources          : @{AdditionalParams=; AllocationOrder=0; CompartmentOperationTime=0; Flags=0; Health=; ID=4503493F-861A-4175-A476-A8ED2F0940A2;
                     PortOperationTime=0; State=1; SwitchOperationTime=0; VfpOperationTime=0; parentId=0A78ECBD-F5C7-4831-80BF-B242B16B191C}
State              : 1
VirtualNetwork     : 76177bd4-a2b5-43a6-9cb8-a37d9d903123
VirtualNetworkName : nat
MacAddress         : 00-15-5D-CD-BE-47
IPAddress          : 172.27.148.101
PrefixLength       : 20
GatewayAddress     : 172.27.144.1
IPSubnetId         : 6815fda3-cd3e-47b7-adfe-ef4d32160abe
DNSServerList      : 192.168.1.15
DNSSuffix          : internal.example.com
Namespace          : @{ID=2c094b9e-af3a-4dc1-97d8-821b68603d23}
SharedContainers   : {}

[github-actions]

Thank you for creating an Issue. Please note that GitHub is not an official channel for Microsoft support requests. To create an official support request, please open a ticket here. Microsoft and the GitHub Community strive to provide a best effort in answering questions and supporting Issues on GitHub.

[riverar]

Closing for now. Appears to be a deficiency of ctr itself.