Only half work seems to be done

[EWagRP]

Hello there again,

I need to seek help, because I step on the same spot over and over again.

Well, I achieved to prepare some bootfs parts, but any given time I fail on the rootfs.

First I´d like to share my config:
CUSTOMER_KEY_FILE_PEM=/home/pi/SECURE-BOOT-Dependency/privatekey.pem
CUSTOMER_KEY_PKCS11_NAME=
GOLD_MASTER_OS_FILE=/home/pi/SECURE-BOOT-Dependency/2025-01-02-raspios-bookworm-armhf.img
RPI_DEVICE_STORAGE_TYPE=sd
RPI_DEVICE_STORAGE_CIPHER=aes-xts-plain64
RPI_DEVICE_FAMILY=5
RPI_DEVICE_BOOTLOADER_CONFIG_FILE=/var/lib/rpi-sb-provisioner/bootloader.config
RPI_DEVICE_LOCK_JTAG=
RPI_DEVICE_EEPROM_WP_SET=
RPI_DEVICE_FETCH_METADATA=
RPI_DEVICE_RETRIEVE_KEYPAIR=
DEMO_MODE_ONLY=1
RPI_SB_WORKDIR=/home/pi/SECUREBOOT
RPI_SB_PROVISONER_MANUFACTURING_DB=

The entire output is too long for this post, but you can find it as attached file:
output_rpi_sb_provisioner.txt

I had to change some lines in rpi-sb-provisioner.sh:
rpi-sb-provisioner.sh.txt

The changes can be found in,
line 343 cleanup(): #rm -rf "${TMP_DIR}" otherwise the device always remains busy and can´t delete the directories either.

line 382 changed SOURCE_EEPROM_IMAGE="/lib/firmware/raspberrypi/bootloader-2712/latest/pieeprom-2025-01-14.bin"
pieeprom-2024-09-23.bin seems to be outdated.

And there is something else I figured out. In function augment_initramfs() my kernel variable is empty, but I wonder why.

Well in the end I got these files in my output folder:

• bootfiles.bin
• bootfs-temporary.img
• boot.img
• boot.sig
• config.txt

My monitor approved the process failed.

keywriter.log
provisioner.log
triage.log

Is there anybody understanding my issue ? :-)
How can I successfully prepare my secure boot?

[tdewey-rpi]

At first glance, the initial configuration looks to be broken:

+ RPI_DEVICE_STORAGE_TYPE=sd
+ RPI_DEVICE_STORAGE_CIPHER=aes-xts-plain64
+ RPI_DEVICE_FAMILY=5

+ RPI_SB_WORKDIR=/home/pi/SECUREBOOT

1. Do not set RPI_SB_WORKDIR while you are iterating on your image. WORKDIR is designed to optimise back-to-back flashing scenarios (mass provisioning).
2. You appear to have selected an armhf build of Raspberry Pi OS, rather than the arm64 build. armhf builds are explicitly not supported at this time - they come from a subtly different package lineage than arm64, and that difference can cause unexpected failures.

I'd expect the kernel variable to be empty if, after looking at the initramfs, we couldn't find any sign of a modules directory. Which would be extremely odd - because we should have created it based on locating the required cryptography modules in the lines above.

The most plausible reason I can think for that search failing is that the armhf image is structured differently - and if that's the case, using arm64 might be exactly the change you need.

[EWagRP]

Hi Tom,

I altered all the things you mentioned, but it didn´t work out.

I have rebuilt the image with pi-gen (this time arm64), added it to my config. And started the process again.
Then I got left with the same issue, kernel variable is still empty.

So I decided to reset the entire system. Downloaded the latest 64bit version of raspberryOS as my foundation. Installed the provisioner via apt install on it. And put all the things together like descibed. - Again nothing in the kernel variable.
Then I tried to make use of the latest 64bit raspberryOS instead of my pi-gen version. - Again nothing in kernel variable.

What else can I try?

Regards Erwin

[tdewey-rpi]

@EWagRP Sounds like we need to investigate further on our side. I'll drop another comment once we understand what's happened.

Tagging @roliver-rpi for visibility.

[EWagRP]

Thanks guys, I really appreciate your help!

[tdewey-rpi]

I suspect that you're seeing a breakage in DEMO_MODE_ONLY, which we had previously introduced to help debug and iterate the project.

Assuming you're ready to actually write the OS - are you prepared to try again with DEMO_MODE_ONLY=?

[EWagRP]

Hi there,

i changed my config to DEMO_MODE_ONLY=
and i´ve bricked four PIs up to now.

I really don´t know what´s going on here, but things are getting really expensive.
You can find my output in the attached file.

fail.txt

Would you mind it to connect to my device some day to figure out why the process fails?

[tdewey-rpi]

Good news: Your devices are not bricks, you just need to provide software that's been signed accordingly.

Bad news: You seem to be running the binary directly, rather than using the systemd services. This is not in any way supported - rpi-sb-provisioner is designed to be a hands-off system that operates happily without any intervention (or monitoring, if you like to live dangerously).

In this case, it looks like you've got a particularly exotic execution environment:

/dev/loop1
/dev/loop0: failed to use device: No such device```

During the modification of the supplied operating system to work correctly in a full-disk-encryption-with-signed-boot world, we use `losetup` and other commands to operate on the image. In this case, it looks like your system may have already set up loop devices - not something `rpi-sb-provisioner` expects from a provisioning system.

Finally, the log was doomed to fail at the start:

```sudo rpi-sb-provisioner.sh 
+ TARGET_DEVICE_SERIAL=```

rpi-sb-provisioner was invoked without any target serial - so it couldn't operate correctly.

--

To get back to a useful state, I'd recommend the following:

- Install `rpi-sb-provisioner` from `apt` on a machine you will use **exclusively** for provisioning
- Make sure you configure it to use the same key that you've written to the devices
- Connect the 'bricked' devices to the provisioning machine by USB

[EWagRP]

Hello again,

:-) this time I come along with a big smile on my face!
I made it, two devices run with secure boot now.

Tomorrow I´ll go on with some more tests. Especially mass deployment will be the thing to catch up with.

Thanks for your patience and support!

I really appreciate your tips:

To get back to a useful state, I'd recommend the following:

• Install rpi-sb-provisioner from apt on a machine you will use exclusively for provisioning
• Make sure you configure it to use the same key that you've written to the devices
• Connect the 'bricked' devices to the provisioning machine by

But these were not the points that messed up my process. Except that little issue when I missed out the serial device name.
Anyway, I´ve reset the entire system again. And got the same results as days before.
But afterwards I realized, that I had to bringt my Pi back to boot-mode again during the provisioning process, at least twice.
So I replugged the USB-C connection and held the reset button for a short while again. - BAAAM!! - The process continued!

The Excitement took over as I saw the rootfs was getting prepared!!
Eventually the whole process finished without any further issues.

Let´s hope I can reproduce that work tomorrow again - keep my fingers crossed.

Let me know if you visit the embedded world in Germany next month. A beer is on me.

[tdewey-rpi]

Hello again,

:-) this time I come along with a big smile on my face! I made it, two devices run with secure boot now.

Tomorrow I´ll go on with some more tests. Especially mass deployment will be the thing to catch up with.

Thanks for your patience and support!

I really appreciate your tips:

To get back to a useful state, I'd recommend the following:

• Install rpi-sb-provisioner from apt on a machine you will use exclusively for provisioning
• Make sure you configure it to use the same key that you've written to the devices
• Connect the 'bricked' devices to the provisioning machine by

But these were not the points that messed up my process. Except that little issue when I missed out the serial device name. Anyway, I´ve reset the entire system again. And got the same results as days before. But afterwards I realized, that I had to bringt my Pi back to boot-mode again during the provisioning process, at least twice. So I replugged the USB-C connection and held the reset button for a short while again. - BAAAM!! - The process continued!

The Excitement took over as I saw the rootfs was getting prepared!! Eventually the whole process finished without any further issues.

Let´s hope I can reproduce that work tomorrow again - keep my fingers crossed.

I think the incoming PR #118 is going to make your day a little bit better, then. It appears the automated retry mechanism I introduced to catch pre-signed devices doesn't work the way I hoped, and so I've added a side-step file you can insert into the device's log area to bypass the keywriting that wouldn't proceed.

Let me know if you visit the embedded world in Germany next month. A beer is on me.