Heads panic (unable to mount root fs) before menu shows up

[marmarek]

Please identify some basic details to help process the report

A. Provide Hardware Details

1. What board are you using? (Choose from the list of boards here)

2. Does your computer have a dGPU or is it iGPU-only?

   • dGPU (Distinct GPU other then internal GPU)
   • iGPU-only (Internal GPU, normally Intel GPU)

3. Who installed Heads on this computer?

   • Insurgo (Issues to be reported at https://github.com/linuxboot/heads/issues)
   • Nitrokey (Issues to be reported at https://github.com/Nitrokey/heads/issues)
   • Purism (Issues to be reported at https://source.puri.sm/firmware/pureboot/-/issues)
   • Novacustom (Issues to be reported at https://github.com/Dasharo/dasharo-issues)
   • HardnenedVault (Issues to be reported at https://github.com/hardenedvault/vaultboot/issues)
   • Other provider
   • Self-installed

4. What PGP key is being used?

   • Librem Key (Nitrokey Pro 2 rebranded)
   • Nitrokey Pro
   • Nitrokey Pro 2
   • Nitrokey 3 NFC
   • Nitrokey 3 NFC Mini
   • Nitrokey Storage
   • Nitrokey Storage 2
   • Yubikey
   • Other

5. Are you using the PGP key to provide HOTP verification?

   • Yes
   • No
   • I don't know

B. Identify how the board was flashed

1. Is this problem related to updating heads or flashing it for the first time?

   • First-time flash
   • Updating heads

2. If the problem is related to an update, how did you attempt to apply the update?

   • Using the Heads menus
   • Flashrom via the Recovery Shell
   • External flashing

3. How was Heads initially flashed?

   • External flashing
   • Internal-only / 1vyprep+1vyrain / skulls
   • Don't know

4. Was the board flashed with a maximized or non-maximized/legacy rom?

   • Maximized
   • Non-maximized / legacy
   • I don't know

5. If Heads was externally flashed, was IFD unlocked?

   • Yes
   • No
   • Don't know

C. Identify the rom related to this bug report

1. Did you download or build the rom at issue in this bug report?

   • I downloaded it
   • I built it

2. If you downloaded your rom, where did you get it from?

   • Heads CircleCi
   • Purism
   • Nitrokey
   • Dasharo DTS (Novacustom)
   • Somewhere else (please identify)

heads-x230-hotp-maximized_usb-kb-v0.2.0-2416-gcd683b1

3. If you built your rom, which repository:branch did you use?

   • Heads:Master
   • Other (please identify)

4. What version of coreboot did you use in building?
   { You can find this information from github commit ID or once flashed, by giving the complete version from Sytem Information under Options --> menu}

5. In building the rom, where did you get the blobs?

   • No blobs required
   • Provided by the company that installed Heads on the device
   • Extracted from a backup rom taken from this device
   • Extracted from another backup rom taken from another device (please identify the board model)
   • Extracted from the online bios using the automated tools provided in Heads
   • I don't know

Please describe the problem

Describe the bug

After one of many reinstalls (this is a CI system), Heads doesn't start anymore. Could be Heads initramfs broken?

To Reproduce
Steps to reproduce the behavior:

1. Reinstall Qubes OS many times, including resetting TPM, regenerating HOTP keys etc
2. ...
3. See error

Expected behavior

No panic

Screenshots

A screen with the message: Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)

Additional context

I can do a flash dump with external programmer (if that would be helpful), but only in a few days.

[tlaurion]

@marmarek a full dump could help, but it looks like something spi got corrupted during internal flash. The error means that initramfs could not be found, after coreboot extracting payload in ram and trying to jump to its initramfs. Seems like spi content got corrupted somehow. Weird anyhow, since internal flashing either used flashrom/flashprog so it it should have failed if incapable of writing a block and should have failed verification step from prior rom version, leading you to recovery shell in case an erase/write operation on spi failed from prior rom flashing spi to new version.

Can you provide source rom version as well if you kept trace? cd683b1 is pretty old (2 months old)? On phone, will get back to this on computer later and inspects diffs, but as said previously flash op should have failed verification step, where rollback to previous rom could have mitigated leaving you in recovery shell in case verifying flashed rom content failing, and leaving you to old rom forever in case spi is worn out for whatever reason.

I'm mostly interested into external spi backup dump content, as well as if your test machine can boot from an externally flashed roms from master, or better, feature freeze PR branch.

Next step is to diagnose if spi chips are worn out. And if not, trying to explain what happened to result in corrupted spi flash otherwise.

[marmarek]

Can you provide source rom version as well if you kept trace? cd683b1 is pretty old (2 months old)? On phone, will get back to this.

From before the update? I don't have it. But it did worked for a while after the update. Last working date is 2024-12-14.

[tlaurion]

@marmarek a full dump could help, but it looks like something spi got corrupted during internal flash. The error means that initramfs could not be found, after coreboot extracting payload in ram and trying to jump to its initramfs. Seems like spi content got corrupted somehow. Weird anyhow, since internal flashing either used flashrom/flashprog so it it should have failed if incapable of writing a block and should have failed verification step from prior rom version, leading you to recovery shell in case an erase/write operation on spi failed from prior rom flashing spi to new version.

Can you provide source rom version as well if you kept trace? cd683b1 is pretty old (2 months old)? On phone, will get back to this on computer later and inspects diffs, but as said previously flash op should have failed verification step, where rollback to previous rom could have mitigated leaving you in recovery shell in case verifying flashed rom content failing, and leaving you to old rom forever in case spi is worn out for whatever reason.

I'm mostly interested into external spi backup dump content, as well as if your test machine can boot from an externally flashed roms from master, or better, feature freeze PR branch.

Next step is to diagnose if spi chips are worn out. And if not, trying to explain what happened to result in corrupted spi flash otherwise.

@marmarek ping because edited post on phone.

[marmarek]

External dump of broken state:
bad-rom.zip

I've dumped it twice and got the same result, so at least it is not very worn out flash...

[marmarek]

better, feature freeze PR branch.

which one specifically?

[tlaurion]

better, feature freeze PR branch.

which one specifically?

#1875