Raspberry Pi 4 SD card image broken on master

[markuskowa]

Describe the bug
The build of the RPI4 image has been broken for a few weeks now. See: https://hydra.nixos.org/job/nixos/trunk-combined/nixos.sd_image_raspberrypi4.aarch64-linux

The module closure fails to build with:

root module: sun4i_drm
modprobe: FATAL: Module sun4i_drm not found in directory /nix/store/86byp9ixv7yy94vaah92i926ff7i0068-linux-5.4.79-1.20201201-modules/lib/modules/5.4.79

CC @samueldr

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute: 
# a list of nixos modules affected by the problem
module: installer/cd-dvd/sd-image-raspberrypi4.nix

[samueldr]

Caused by #78430

See #109280

[samueldr]

For the time being, there is no quick way to work around the issue. From the list of available modules, most are needed. We'd need to be able to filter out a module from the list, which AFAIK for lists is impossible with NixOS modules.

So uh... annoyingly it'll have to stay this way until some resolution of the overall issue :/...

An alternative, but quite intrusive, would be to split the aarch64 profile into an "aarch64 base profile", from which the normal aarch64 and the raspberry pi 4 profiles inherit. Though I don't really like the approach since the raspberry pi 4 profile is, overall, a temporary measure for a somewhat popular and hard-to-work-with platform.

It might be more worthwhile , in the long run, to take a final look at 5.10/5.11 mainline and if it is enough to allow a user to bootstrap themselves finally.

[domenkozar]

@samueldr couldn't we allow missing kernel modules when generating images at least?

[samueldr]

The interface/abstraction for availableKernelModules is lacking. The fact it's a list, and lists are really hard to manipulate once set in NixOS modules makes this basically a hard task. I guess augmenting #78430 with a (default true) failOnMissingModule would help. But really what would be required is to tell per module whether it is needed, and allow overriding data.

So availableKernelModules needs to somehow allow setting "yes/no/optional" per-module, in an ideal world.

[mohe2015]

Quick idea but what about adding a optionalKernelModules or something like that and then use availableKernelModules \setminus optionalKernelModules as the required kernel modules?

Edit: or yeah depending on how it would work using a dictionary?

[samueldr]

Using an attrset for availableKernelModules would be the solution. But that breaks existing use cases by default, probably.

There might be something coming to handle this issue

• https://logs.nix.samueldr.com/nixos-dev/2021-05-05#1620236254-1620237510;

But this is a hard problem to solve correctly AFAIUI.

I wouldn't like having the bad abstraction any more than the current breakage that I don't like.

[domenkozar]

I suggest then to revert the commit that broke this until we have a better abstraction.

[domenkozar]

I've merged #121720 that fixes this issue.

Once it reaches the channels you can download it via Hydra.

You can build it on nixpkgs master using nix-build -A sd_image_new_kernel.aarch64-linux nixos/release.nix.

[samueldr]

I don't think it solved this issue.

#121834 works around the issue by removing the raspberry pi 4 board-specific image.

With #121720 merged, the modules are still added to the raspberry pi 4 board-specific image, even if they're not present in the kernel build. In fact, there may be more failures! It ends up adding more modules present in mainline, absent from the vendor-specific kernel build.

[domenkozar]

https://nixos.wiki/wiki/NixOS_on_ARM/Raspberry_Pi_4 recommends using the generic image anyway, so that's why I considered it fixed.

[samueldr]

Ah sorry, I pre-emptively edited this page. And I'm splitting hair right now because technically the issue is still going on. But really it's all but merged-and-solved.

[necrophcodr]

It appears to still be broken. Booting the latest image from Hydra ( https://hydra.nixos.org/build/142828023 ) results in the image being unable to find /nix/.../init, and failing to continue booting from there.

[domenkozar]

@necrophcodr I've booted the same image and it works. What steps did you take?

Mine are:

  $ nix-shell -p wget zstd
  $ wget https://hydra.nixos.org/build/142828023/download/1/nixos-sd-image-21.05pre288297.8eed0e20953-aarch64-linux.img.zst
  $ unzstd -d nixos-sd-image-21.05pre288297.8eed0e20953-aarch64-linux.img.zst
  # sudo dd if=nixos-sd-image-21.05pre288297.8eed0e20953-aarch64-linux.img of=/dev/sdX bs=4096 conv=fsync status=progress

[necrophcodr]

I performed the same steps, except the last one where I instead used GNOME Disks to write the disk image to the SD card. I don't have a way to reproduce it currently, since the Raspberry Pi 4 I was testing it on, has since been formatted with a different OS to be used in a production system tomorrow. I'll see if I can get my hands on another RPi 4B and perform the same steps again.

edit

With an exception that I used unzstd instead.

[domenkozar]

Same, unzip was a typo.

[necrophcodr]

I've gotten my hands on one now for a few minutes, and it appears to be getting an error like:

mmc1: error -5 whilst initialising SD card

It prints this continually, until I press a key, and are presented with the option to reboot or continue. Continuing kernel panics, and reboot, well, reboots it.

[necrophcodr]

The SD card used is the NOOBs SD card, but overwritten using the dd command as above, but with the of device replaced.

[necrophcodr]

Booting with the same SD card using Ubuntu Server 64bit OS or Ubuntu Core works fine.

[samueldr]

It sounds like it "works" up until stage-1, where the kernel is somehow unable to work with the SD card. It might be a combination of [ "sd card" "sd reader" "linux version" ] that is causing the issue. Do you have other brands SD cards, or same brand but (probably) different batches?

It would be helpful to know whether "Ubuntu Server 64bit OS" and "Ubuntu Core" are using the mainline kernel or not. The driver differences might exacerbate issues on marginal hardware.

[samueldr]

The underlying issue is now invalid with #121834 being merged.

@necrophcodr I invite you to open an issue, linking to this issue, for your particular problem. It seems you have some kind of hardware-specific issue at first glance.

[necrophcodr]

Thanks, i will test it with other SD cards as well. It appears to only happen with this distribution, but it could certainly still be related to the hardware.

[samueldr]

It is likely the other distributions are using the vendor kernel, and that the driver differs in a way where only mainline has a specific issue. Using the "latest kernel" variant of the image may also be something to look into.

[necrophcodr]

It is likely the other distributions are using the vendor kernel, and that the driver differs in a way where only mainline has a specific issue. Using the "latest kernel" variant of the image may also be something to look into.

I've tried the latest kernel variant as well, but I'll create a new issue with more details once I've collected more information.

[anandsuresh]

I am trying to build NixOS for the raspberry Pi 4 and ran into the same error that started this thread. FWIW, I'm seeing this on nixpkgs 21.05-release and on master as well.

Would it be prudent to re-open this ticket or file a fresh issue?

[samueldr]

@anandsuresh please open a new issue and ping this issue with the specific problem you're observing detailed. Please detail what you are attempting to do (e.g. whether it's booting through SD or USB), and as much information as you can.

Note that there is a known issue (totally unrelated to this one) about USB boot, (#143885).