Intel wifi broken since 6.2 (iwlwifi) on AC 9560

[brokentoaster]

I have an Intel(R) Wireless-AC 9560 160MHz, REV=0x354 that has been working fine but is now broken on all versions since 6.2. I've attached logs of good and bad below. Let me know if there is anything more I can do to help. I'm happy to do some testing if needed or just carry on using v6.2

Running on Freebsd 13.0-RELEASE

wifibox_fail_8.0.tar.gz
wifibox_good_6.2.tar.gz

The dmesg in 6.2 was as follows.

--->8---
[    1.167689] Intel(R) Wireless WiFi driver for Linux
[    1.167797] iwlwifi 0000:00:06.0: can't derive routing for PCI INT A
[    1.167799] iwlwifi 0000:00:06.0: PCI INT A: no GSI
[    1.169177] iwlwifi 0000:00:06.0: Direct firmware load for iwlwifi-QuZ-a0-jf-b0-66.ucode failed with error -2
[    1.169224] iwlwifi 0000:00:06.0: Direct firmware load for iwlwifi-QuZ-a0-jf-b0-65.ucode failed with error -2
[    1.169276] iwlwifi 0000:00:06.0: Direct firmware load for iwlwifi-QuZ-a0-jf-b0-64.ucode failed with error -2
[    1.171401] iwlwifi 0000:00:06.0: api flags index 2 larger than supported by driver
[    1.171412] iwlwifi 0000:00:06.0: TLV_FW_FSEQ_VERSION: FSEQ Version: 89.3.35.37
[    1.171602] iwlwifi 0000:00:06.0: loaded firmware version 63.c04f3485.0 QuZ-a0-jf-b0-63.ucode op_mode iwlmvm
[    1.172491] e1000: Intel(R) PRO/1000 Network Driver
[    1.172492] e1000: Copyright (c) 1999-2006 Intel Corporation.
[    1.216237] iwlwifi 0000:00:06.0: Detected Intel(R) Wireless-AC 9560 160MHz, REV=0x354
[    1.234033] thermal thermal_zone0: failed to read out thermal zone (-61)
[    1.337214] iwlwifi 0000:00:06.0: Detected RF JF, rfid=0x10xxxx
[    1.400176] iwlwifi 0000:00:06.0: base HW address: 74:d8:xx:xx:xx:xx
--->8---

Now the iwlwifi fails to load with the following messages in dmesg.

--->8---
   2.776643] iwlwifi 0000:00:06.0: Detected Intel(R) Wireless-AC 9560 160MHz, REV=0x354
[    2.789780] thermal thermal_zone0: failed to read out thermal zone (-61)
[    2.905811] NET: Registered PF_PACKET protocol family
[    3.803176] iwlwifi 0000:00:06.0: SecBoot CPU1 Status: 0x57af, CPU2 Status: 0x3
[    3.803227] iwlwifi 0000:00:06.0: UMAC PC: 0x80xxxxxx
[    3.803275] iwlwifi 0000:00:06.0: LMAC PC: 0x4bxxxx
[    3.803284] iwlwifi 0000:00:06.0: WRT: Collecting data: ini trigger 13 fired (delay=0ms).
[    3.803484] iwlwifi 0000:00:06.0: Start IWL Error Log Dump:
[    3.803487] iwlwifi 0000:00:06.0: Transport status: 0x00000042, valid: 6
[    3.803490] iwlwifi 0000:00:06.0: Loaded firmware version: 63.c04f3485.0 QuZ-a0-jf-b0-63.ucode
[    3.803493] iwlwifi 0000:00:06.0: 0x00000084 | NMI_INTERRUPT_UNKNOWN       
[    3.803497] iwlwifi 0000:00:06.0: 0x000022F0 | trm_hw_status0
[    3.803500] iwlwifi 0000:00:06.0: 0x00000000 | trm_hw_status1
[    3.803502] iwlwifi 0000:00:06.0: 0x004C2A82 | branchlink2
[    3.803505] iwlwifi 0000:00:06.0: 0x004B9C26 | interruptlink1
[    3.803507] iwlwifi 0000:00:06.0: 0x004B9C26 | interruptlink2
[    3.803509] iwlwifi 0000:00:06.0: 0x004C180A | data1
[    3.803511] iwlwifi 0000:00:06.0: 0x01000000 | data2
[    3.803514] iwlwifi 0000:00:06.0: 0x00000000 | data3
[    3.803516] iwlwifi 0000:00:06.0: 0x00000000 | beacon time
[    3.803518] iwlwifi 0000:00:06.0: 0x000F1B8B | tsf low
[    3.803521] iwlwifi 0000:00:06.0: 0x00000000 | tsf hi
[    3.803523] iwlwifi 0000:00:06.0: 0x00000000 | time gp1
[    3.803525] iwlwifi 0000:00:06.0: 0x000F7529 | time gp2
[    3.803527] iwlwifi 0000:00:06.0: 0x00000001 | uCode revision type
[    3.803530] iwlwifi 0000:00:06.0: 0x0000003F | uCode version major
[    3.803532] iwlwifi 0000:00:06.0: 0xC04F3485 | uCode version minor
[    3.803534] iwlwifi 0000:00:06.0: 0x00000351 | hw version
[    3.803537] iwlwifi 0000:00:06.0: 0x18489004 | board version
[    3.803539] iwlwifi 0000:00:06.0: 0x8004F502 | hcmd
[    3.803541] iwlwifi 0000:00:06.0: 0x00020000 | isr0
[    3.803543] iwlwifi 0000:00:06.0: 0x00000000 | isr1
[    3.803546] iwlwifi 0000:00:06.0: 0x00F00002 | isr2
[    3.803548] iwlwifi 0000:00:06.0: 0x00C0000C | isr3
[    3.803550] iwlwifi 0000:00:06.0: 0x00000000 | isr4
[    3.803552] iwlwifi 0000:00:06.0: 0x00000000 | last cmd Id
[    3.803554] iwlwifi 0000:00:06.0: 0x004C180A | wait_event
[    3.803556] iwlwifi 0000:00:06.0: 0x00000000 | l2p_control
[    3.803559] iwlwifi 0000:00:06.0: 0x00000000 | l2p_duration
[    3.803561] iwlwifi 0000:00:06.0: 0x00000000 | l2p_mhvalid
[    3.803563] iwlwifi 0000:00:06.0: 0x00000000 | l2p_addr_match
[    3.803565] iwlwifi 0000:00:06.0: 0x0000000B | lmpm_pmg_sel
[    3.803567] iwlwifi 0000:00:06.0: 0x00000000 | timestamp
[    3.803569] iwlwifi 0000:00:06.0: 0x00B80000 | flow_handler
[    3.803629] iwlwifi 0000:00:06.0: Start IWL Error Log Dump:
[    3.803631] iwlwifi 0000:00:06.0: Transport status: 0x00000042, valid: 7
[    3.803634] iwlwifi 0000:00:06.0: 0x20000066 | NMI_INTERRUPT_HOST
[    3.803637] iwlwifi 0000:00:06.0: 0x00000000 | umac branchlink1
[    3.803639] iwlwifi 0000:00:06.0: 0x804530BA | umac branchlink2
[    3.803642] iwlwifi 0000:00:06.0: 0x8047117E | umac interruptlink1
[    3.803644] iwlwifi 0000:00:06.0: 0x8047117E | umac interruptlink2
[    3.803646] iwlwifi 0000:00:06.0: 0x01000000 | umac data1
[    3.803649] iwlwifi 0000:00:06.0: 0x8047117E | umac data2
[    3.803651] iwlwifi 0000:00:06.0: 0x00000000 | umac data3
[    3.803653] iwlwifi 0000:00:06.0: 0x0000003F | umac major
[    3.803664] iwlwifi 0000:00:06.0: 0xC04F3485 | umac minor
[    3.803667] iwlwifi 0000:00:06.0: 0x000F7527 | frame pointer
[    3.803669] iwlwifi 0000:00:06.0: 0xC0886270 | stack pointer
[    3.803671] iwlwifi 0000:00:06.0: 0x00000000 | last host cmd
[    3.803673] iwlwifi 0000:00:06.0: 0x00000000 | isr status reg
[    3.803716] iwlwifi 0000:00:06.0: IML/ROM dump:
[    3.803718] iwlwifi 0000:00:06.0: 0x00000003 | IML/ROM error/state
[    3.803764] iwlwifi 0000:00:06.0: 0x000057AF | IML/ROM data1
[    3.803795] iwlwifi 0000:00:06.0: 0x00000080 | IML/ROM WFPM_AUTH_KEY_0
[    3.803840] iwlwifi 0000:00:06.0: Fseq Registers:
[    3.803844] iwlwifi 0000:00:06.0: 0x60000000 | FSEQ_ERROR_CODE
[    3.803863] iwlwifi 0000:00:06.0: 0x80260000 | FSEQ_TOP_INIT_VERSION
[    3.803868] iwlwifi 0000:00:06.0: 0x00020006 | FSEQ_CNVIO_INIT_VERSION
[    3.803887] iwlwifi 0000:00:06.0: 0x0000A384 | FSEQ_OTP_VERSION
[    3.803892] iwlwifi 0000:00:06.0: 0xF1B1685E | FSEQ_TOP_CONTENT_VERSION
[    3.803911] iwlwifi 0000:00:06.0: 0x4552414E | FSEQ_ALIVE_TOKEN
[    3.803915] iwlwifi 0000:00:06.0: 0x20000302 | FSEQ_CNVI_ID
[    3.803935] iwlwifi 0000:00:06.0: 0x01300202 | FSEQ_CNVR_ID
[    3.803939] iwlwifi 0000:00:06.0: 0x20000302 | CNVI_AUX_MISC_CHIP
[    3.803961] iwlwifi 0000:00:06.0: 0x01300202 | CNVR_AUX_MISC_CHIP
[    3.804067] iwlwifi 0000:00:06.0: 0x0000485B | CNVR_SCU_SD_REGS_SD_REG_DIG_DCDC_VTRIM
[    3.804114] iwlwifi 0000:00:06.0: 0xA5A5A5A2 | CNVR_SCU_SD_REGS_SD_REG_ACTIVE_VDIG_MIRROR
[    3.804134] iwlwifi 0000:00:06.0: Failed to start RT ucode: -110
[    3.804138] iwlwifi 0000:00:06.0: WRT: Collecting data: ini trigger 13 fired (delay=0ms).
[    4.423370] iwlwifi 0000:00:06.0: Failed to run INIT ucode: -110
[    4.444905] iwlwifi 0000:00:06.0: retry init count 1
[    4.444965] iwlwifi 0000:00:06.0: Detected Intel(R) Wireless-AC 9560 160MHz, REV=0x354
--->8---

[pgj]

This seems to be related to Linux 5.15.16 and later versions, which was shipped with versions after 0.6.2 (Linux 5.15.5). I have found some similar trouble reports, but all it might need is a more up-to-date set of firmware files. Alpine has not provided an update for the linux-firmware packages, therefore it might be worth to experiment with doing this manually.

I will try to build a version of the disk image with more recent firmware files to see if this can solve the problem.

[pgj]

Here I published a version with the latest Linux firmware files (20220310) that you can try if you want: https://github.com/pgj/freebsd-wifibox-port/tree/c4adb6d0280bfb1007abf6aecaf4c4addd06db75

[brokentoaster]

Thanks, but no change on my machine with that version. Similar issues also seen here on Arch . I'll stick with 6.2 for now. I don't see this as a bug in wifibox so happy to close the issue.

[pgj]

@brokentoaster in the Arch issue you linked, there was a solution mentioned: all the firmware versions above 63 should be removed. It would be interesting to learn if this could fix the problem for you. Could you please try it and report the results?

You can do this by the following commands on the Wifibox console, for example:

mount -o remount,rw /
rm -f $(find /lib/firmware/ | grep -E "iwlwifi-QuZ-a0-jf-b0-(66|68|71).ucode")

And then restart the guest (on the host):

wifibox restart guest

[brokentoaster]

@pgj sadly this didn't fix the issue. I even tried removing v63 as well to check that it was changing and didn't help.
I did notice that when switching from 6.2 back to your 8.2 build it worked fine as the wifi card firmware was already loaded. This didn't persist over a power cycle.

Happy to keep testing if you have more ideas.

[pgj]

Linux 5.15.33 has just been released with a couple of iwlwifi fixes, maybe that is something we can benefit from. I will create an image with this kernel version that you can test.

[pgj]

Here is the version of the port that uses an image based on Alpine 3.15.4 and Linux 5.15.33: https://github.com/pgj/freebsd-wifibox-port/tree/ba978b11f787e93f591fbb95356eaa585e90850c

[brokentoaster]

wifibox:~# uname -a
Linux wifibox 5.15.33-0-lts #1-Alpine SMP Sun, 10 Apr 2022 00:16:48 +0000 x86_64 Linux

Still not working and loading v63

 [    2.068365] iwlwifi 0000:00:06.0: Failed to start RT ucode: -110
...
 [    3.751182] iwlwifi 0000:00:06.0: Loaded firmware version: 63.c04f3485.0 QuZ-a0-jf-b0-63.ucode

[pgj]

Okay. I created an image with Linux 5.10.110, perhaps stepping back a bit can help: https://github.com/pgj/freebsd-wifibox-port/tree/a03e91d0f3e8201847859d6186720a7f2334d503

[pgj]

@brokentoaster ?

[brokentoaster]

Sorry @pgj I missed your post. No luck with that one either.

wifibox:~# uname -a
Linux wifibox 5.10.110-0-lts #1-Alpine SMP Sun, 10 Apr 2022 09:47:06 +0000 x86_64 Linux

[    3.753989] iwlwifi 0000:00:06.0: Failed to run INIT ucode: -110
[    3.766831] iwlwifi 0000:00:06.0: retry init count 1
[    3.767085] iwlwifi 0000:00:06.0: Detected Intel(R) Wireless-AC 9560 160MHz, REV=0x354
[    4.805592] iwlwifi 0000:00:06.0: SecBoot CPU1 Status: 0x5354, CPU2 Status: 0x3
[    4.805637] iwlwifi 0000:00:06.0: UMAC PC: 0x80474fe8
[    4.805685] iwlwifi 0000:00:06.0: LMAC PC: 0x4ba86e
[    4.805690] iwlwifi 0000:00:06.0: WRT: Collecting data: ini trigger 13 fired.
[    4.805890] iwlwifi 0000:00:06.0: Start IWL Error Log Dump:
[    4.805893] iwlwifi 0000:00:06.0: Status: 0x00000000, count: 6
[    4.805895] iwlwifi 0000:00:06.0: Loaded firmware version: 59.601f3a66.0 QuZ-a0-jf-b0-59.ucode

I'm going to boot into alpine 3.15 directly and have a play, see if that reveals anything.

[brokentoaster]

Booting into an Alpine 3.15.4 install with kernel 5.15.32 seemed to work

[    2.501055] iwlwifi 0000:00:14.3: Direct firmware load for iwlwifi-QuZ-a0-jf-b0-66.ucode failed with error -2
[    2.501070] iwlwifi 0000:00:14.3: Direct firmware load for iwlwifi-QuZ-a0-jf-b0-65.ucode failed with error -2
[    2.501268] iwlwifi 0000:00:14.3: Direct firmware load for iwlwifi-QuZ-a0-jf-b0-64.ucode failed with error -2
[    2.504575] iwlwifi 0000:00:14.3: api flags index 2 larger than supported by driver
[    2.504596] iwlwifi 0000:00:14.3: TLV_FW_FSEQ_VERSION: FSEQ Version: 89.3.35.37
[    2.504961] iwlwifi 0000:00:14.3: loaded firmware version 63.c04f3485.0 QuZ-a0-jf-b0-63.ucode op_mode iwlmvm
[    2.532380] iwlwifi 0000:00:14.3: Detected Intel(R) Wireless-AC 9560 160MHz, REV=0x354
[    2.643075] iwlwifi 0000:00:14.3: Detected RF JF, rfid=0x105110
[    2.699163] iwlwifi 0000:00:14.3: base HW address: 74:d8:3e:8b:0a:10

I'll give it another reboot to make sure it works from a cold start.
dmesg_good.log

[pgj]

In that case, it might be a problem related to the bhyve or even its PCI pass-through implementation. There have been many important changes in that area since the release of 13.0-RELEASE, therefore it might be worth a try with a recent 13-STABLE or a release candidate.

[brokentoaster]

OK I'll try an upgrade to 13.1RC2 and see how that goes.

[brokentoaster]

Quick Update. RC13.2 adds support for iwlwifi drivers and so my card now works without wifibox.
The current FreeBSD native is only 802.11g so its not the fastest, so I believe there still a use case for wifibox.

Currently the wifibox bhyve vm will not start, so I suspect I need to disable the iwlwifi module as it will be clashing with PCI pass-through.

Launching wifibox.....ERROR: Guest could not be started.

I'll let you know how I get on over the next couple of weeks, and probably move to RC3 and then 13.1 RELEASE.

[pgj]

Thanks for the update. I have recently added some hints to the manual page on how to stop the native driver loading via using devmatch. Basically, these lines have to be added for /etc/rc.conf:

devmatch_enable="YES"
devmatch_blocklist="if_iwm if_iwlwifi"

Besides that, I would recommend you to enable logging in $ETCDIR/wifibox/core.conf and check (share) the logs from /var/log/wifibox.log to see why the guest could not start up.

[brokentoaster]

Everything seems to work fine now with

wifibox version 0.8.1
Disk image checksum: 49452bd58f610d58b2a04f9450c3eeecf0f5958bd0a0c94334edfc8b2a0545d4

[pgj]

Okay, thank for checking this. Based on that, it indeed seems to be a bhyve issue. Such problems should be tackled by the bhyve+ (as a shim layer). Currently, that is mandatory for FreeBSD 12 (due to the lack for VirtFS support), but due to these changes in the Linux kernel, now is getting to be requirement for FreeBSD 13.0-RELEASE too. Unfortunately, bhyve+ does not yet include the recent bhyve PCI pass-through fixes, thus one has to either move to a recent 13-STABLE or wait until I get there.

That said, we can say that this is not bug in Wifibox itself but a deficiency in bhyve+. I created pgj/freebsd-bhyve-plus-port#3 to track this. On this side, a solution might be to add a note about this under the Caveats section of the documentation or something along that line.

[pgj]

I implemented my proposal in 14c89b9 if you want to check it @brokentoaster .

[pgj]

I have added the commit to main that I mentioned above. It will be released as part of 0.9 but I will also do a patch-level release for 0.8 that will have this.

[pgj]

Based on the comments, I consider this ticket closed.