Odroid C2 | Does not boot on reboot without power cycling

[yandritos]

DietPi version |
G_DIETPI_VERSION_CORE=8
G_DIETPI_VERSION_SUB=1
G_DIETPI_VERSION_RC=2
G_GITBRANCH='master'
G_GITOWNER='MichaIng'
G_LIVE_PATCH_STATUS[0]='applied'
G_LIVE_PATCH_STATUS[1]='applied'
G_LIVE_PATCH_STATUS[2]='not applicable'
G_LIVE_PATCH_STATUS[3]='not applicable'

Distro version |
bullseye

Kernel version |
Linux DietPi 5.10.81-meson64 #21.08.6 SMP PREEMPT Mon Nov 22 11:21:51 UTC 2021 aarch64 GNU/Linux

SBC model |
Odroid C2 (aarch64)

Power supply used |
5V 2.1A

SDcard used |
SanDisk ultra 16Gb

Additional Information (if applicable)

Was the software title installed freshly or updated/migrated?
-Fresh install
Can this issue be replicated on a fresh installation of DietPi?
Yes

Bug report ID | echo $G_HW_UUID
Not applies

Steps to reproduce
-Whenever you send a reboot comand after shooting down, the Odroid C2 never boots up, I need to unplug/plug manually the power suply. (No way to work remotely after a reboot command).
-After plug an screen to the Odroid C2 board and send even locally the reboot, I can see how the OS shut down properly finishing all the tasks, then a black screen and never boots again unles I unplug the board (this never happend with older verions).

Expected behaviour
-A regular restart of the OS.

Actual behaviour
-The board never boots again after a reboot command.

[MichaIng]

Many thanks for your report.

Do you know whether there was a kernel upgrade done? I.e. are the modules in /lib/modules newer than v5.10.81? On a quick look at the Armbian APT repository it seems like it is still the newest "current" kernel, but probably the repos are still syncing.

If a kernel upgrade was done, can you try to flash the new U-Boot:

. /usr/lib/u-boot/platform_install.sh
write_uboot_platform "$DIR" "$(lsblk -npo PKNAME "$G_ROOTFS_DEV")"

[yandritos]

Hi MichaIng,

Thanks a lot for the very fast reply. I executed both commands without errors but after that the result is the same.

I checked the version from the dietpi-config advanced options. --> The current version is Linux 5.10.102-meson64

[MichaIng]

Okay, so probably the kernel upgrade from 5.10.81 to 5.10.102 is responsible. If you have a spare card/drive, could you try it with the current Armbian image, please: https://dl.armbian.com/odroidc2/Bullseye_current
If so, it should be reported here: https://forum.armbian.com/forum/16-amlogic-s905x-s922x/

Btw, how long did you wait before power cycling? Probably there is one service/unit hanging on shutdown. A maximum timeout for any hanging ExecStop command or such should be 90 seconds.

[yandritos]

Hi, I waited even 10 mins without luck. When I plugged the screen on the hdmi port I saw the job with the counter for 1min 30 secs stopping other tasks, that counter finishes even earlier but when all is halted and I get the black screen the new boot never happens.

I have another SD card so today I'll try to use the iso of your link "[Bullseye_current]" (The web SSL cert expired a couple of days ago :D). I'll be back asap with the results.

I have a similar error as commented by a user of an Odroid C4 in your second link:
https://forum.armbian.com/topic/19340-odroid-c4-will-not-reboot-after-any-sort-of-kernel-update-have-tried-running-nand-sata-install/

Thanks for your help!

[yandritos]

Well, confirmed, is a kernel topic. After installing armbian I have the same problem. I reported the isue as you suggested --> https://forum.armbian.com/topic/20241-odroid-c2-never-starting-after-a-soft-reboot-reboot-command/

The universe is telling me to go back to my dietpi from Sep 2021 :D the best option for my Odroid C2 rightnow (preserving the possibility to have the fs-boot in an external drive).

Thanks again for the help!

[MichaIng]

Could you try it with the edge kernel? At least the automated smoke tests, which include a restart test, go through with it:

apt install linux-image-edge-meson64 linux-dtb-edge-meson64 linux-u-boot-odroidc2-edge

[yandritos]

Hi MichaIng,

Sorry I don't want to spend more of your time and mine. I went back finally to the last iso I had from Sep 2021 and I got back the option to run the "root fs" from my external usb ssd. It is working like a charm so I don't really need to run the last version.

Thanks for your help and time!

[yandritos]

I'll continue with the last year version untill a real/relevant reason avoid me to do that :)

[MichaIng]

Let me reopen the issue to keep track of it, since staying on old kernel with vulnerabilities like dirty pipe is not an option for the image we provide.

[MichaIng]

Cross-linking a forum report: REMOVED OBSOLETE LINK

[yandritos]

Let me reopen the issue to keep track of it, since staying on old kernel with vulnerabilities like dirty pipe is not an option for the image we provide.

Hi Michalng sorry for the delay, I think I'll be able to do this with another sd-card during the next days. Do you need me to execute the command "apt install linux-image-edge-meson64 linux-dtb-edge-meson64 linux-u-boot-odroidc2-edge" using the last dietpi release or the current Armbian image you requested before?

[MichaIng]

Good question. I suggest to test it with a fresh Armbian image first, so we can report results on their forums as well. If this works, then it should do so on DietPi as well.

[yandritos]

Hi, after executing the command over the fresh installation of the last armbian you linked, some packages are not found.

Maybe this is easy to solve but my knowledge is very little. Please see below the traces:

---

Welcome to Armbian 22.02.1 with Linux 5.10.102-meson64

No end-user support: community creations

System load: 36% Up time: 0 min
Memory usage: 6% of 1.89G IP: 192.168.1.111
CPU temp: 27°C Usage of /: 2% of 117G

Last login: Thu Apr 14 14:11:29 2022 from 192.168.1.95
root@odroidc2:~# apt install linux-image-edge-meson64 linux-dtb-edge-meson64 linux-u-boot-odroidc2-edge
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
E: Unable to locate package linux-image-edge-meson64
E: Unable to locate package linux-dtb-edge-meson64
E: Unable to locate package linux-u-boot-odroidc2-edge_

[MichaIng]

Did you run apt update already?

[yandritos]

Hi Michalng,

This is after apt update:

---

Fetched 112 MB in 48s (2,333 kB/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
44 packages can be upgraded. Run 'apt list --upgradable' to see them.
N: Repository 'http://deb.debian.org/debian bullseye InRelease' changed its 'Ver                                     sion' value from '11.2' to '11.3'
root@odroidc2:~# apt install linux-image-edge-meson64 linux-dtb-edge-meson64 lin                                     ux-u-boot-odroidc2-edge
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages will be REMOVED:
  linux-u-boot-odroidc2-current
The following NEW packages will be installed:
  linux-dtb-edge-meson64 linux-image-edge-meson64 linux-u-boot-odroidc2-edge
0 upgraded, 3 newly installed, 1 to remove and 44 not upgraded.
Need to get 55.1 MB of archives.
After this operation, 99.7 MB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 https://es-mirror.bret.dk/armbian/apt bullseye/main arm64 linux-dtb-edge-m                                     eson64 arm64 22.05.0-trunk.0038 [313 kB]
Get:3 https://es-mirror.bret.dk/armbian/apt bullseye/main arm64 linux-u-boot-odr                                     oidc2-edge arm64 22.02.1 [307 kB]
Err:2 https://armbian.site-meganet.com/apt bullseye/main arm64 linux-image-edge-                                     meson64 arm64 22.05.0-trunk.0038
  404  Not Found [IP: 188.165.219.168 443]
Fetched 619 kB in 1s (647 kB/s)
E: Failed to fetch https://armbian.site-meganet.com/apt/pool/main/l/linux-5.17.1                                     -meson64/linux-image-edge-meson64_22.05.0-trunk.0038_arm64.deb  404  Not Found [                                     IP: 188.165.219.168 443]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-mis                                     sing?

---

Then, as per the error recommendation I added "--fix-mis". This time it was installed, this was the output

---

root@odroidc2:~# apt install linux-image-edge-meson64 linux-dtb-edge-meson64 linux-u-boot-odroidc2-edge
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages will be REMOVED:
  linux-u-boot-odroidc2-current
The following NEW packages will be installed:
  linux-dtb-edge-meson64 linux-image-edge-meson64 linux-u-boot-odroidc2-edge
0 upgraded, 3 newly installed, 1 to remove and 44 not upgraded.
Need to get 54.5 MB/55.1 MB of archives.
After this operation, 99.7 MB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 https://es-mirror.bret.dk/armbian/apt bullseye/main arm64 linux-image-edge-meson64 arm64 22.05.0-trunk.0038 [54.5 MB]
Fetched 54.5 MB in 9s (6,271 kB/s)
(Reading database ... 41072 files and directories currently installed.)
Removing linux-u-boot-odroidc2-current (22.02.1) ...
Selecting previously unselected package linux-dtb-edge-meson64.
(Reading database ... 41066 files and directories currently installed.)
Preparing to unpack .../linux-dtb-edge-meson64_22.05.0-trunk.0038_arm64.deb ...
Unpacking linux-dtb-edge-meson64 (22.05.0-trunk.0038) ...
Selecting previously unselected package linux-image-edge-meson64.
Preparing to unpack .../linux-image-edge-meson64_22.05.0-trunk.0038_arm64.deb ...
Unpacking linux-image-edge-meson64 (22.05.0-trunk.0038) ...
Selecting previously unselected package linux-u-boot-odroidc2-edge.
Preparing to unpack .../linux-u-boot-odroidc2-edge_22.02.1_arm64.deb ...
Unpacking linux-u-boot-odroidc2-edge (22.02.1) ...
Setting up linux-dtb-edge-meson64 (22.05.0-trunk.0038) ...
Setting up linux-image-edge-meson64 (22.05.0-trunk.0038) ...
dkms: running auto installation service for kernel 5.17.1-meson64:.
update-initramfs: Generating /boot/initrd.img-5.17.1-meson64
update-initramfs: Converting to u-boot format
Setting up linux-u-boot-odroidc2-edge (22.02.1) ...

---

After that the result is the same, I execute a reboot and no effect. When is down the led remains in red and never again boots until I unplug and re-plug the power suply.

[yandritos]

Even I tried a reínstall, but already detected as the last version. Never working after the reboot.

root@odroidc2:~# apt install linux-image-edge-meson64 linux-dtb-edge-meson64 linux-u-boot-odroidc2-edge
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
linux-dtb-edge-meson64 is already the newest version (22.05.0-trunk.0038).
linux-image-edge-meson64 is already the newest version (22.05.0-trunk.0038).
linux-u-boot-odroidc2-edge is already the newest version (22.02.1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

[MichaIng]

Can you again try to flash U-Boot after installing the edge packages:

. /usr/lib/u-boot/platform_install.sh
write_uboot_platform "$DIR" "$(lsblk -npo PKNAME "$G_ROOTFS_DEV")"

It is strange that neither the smoke tests, nor the one user (with Debian Buster) are able to replicate the issue. I wonder whether it may be due to insufficient voltage/power, other due to other attached peripherals/hardware. I see you have a 2.1A PSU, which from the numbers is sufficient: https://wiki.odroid.com/odroid-c2/hardware/hardware#specifications

The ODROID-C2 consumes less than 0.8A in most cases, but it can climb to 2A if many passive USB peripherals are attached directly to the main board.

But PSUs as well as USB cables have a high quality range, not all PSUs are really capable of reliably provide the 5V voltage with tight tolerance, especially when being actually a charger and not for driving power consumers.

[yandritos]

Hi,

I am thinking that the power suply is not the problem. In the other sdcard I have my old dietpi released on sep 2021 with an additional ssd unit connected via USB (not additional power suply for the ssd unit) working like a charm and rebooting without any problem, even that deitpi is a debian buster one if I remember well.

I executed the commands requested, this is the output (note "lsblk: : not a block device"):

Note: for these tests I don't have connected the external ssd drive via usb.

---

Last login: Thu Apr 14 16:16:24 2022 from 192.168.1.95
root@odroidc2:~# . /usr/lib/u-boot/platform_install.sh
root@odroidc2:~# write_uboot_platform "$DIR" "$(lsblk -npo PKNAME "$G_ROOTFS_DEV")"
lsblk: : not a block device
root@odroidc2:~# 

[MichaIng]

even that deitpi is a debian buster one if I remember well.

That is interesting, the user on the Armbian forum who wasn't able to replicate as well ran a Buster image. I have no idea how the userland should affect it, since AFAIK, the kernel, dtb and U-Boot builds are exactly the same. But an idea for further debugging the userland part would be to enable persistent system logs:

dietpi-software uninstall 103
mkdir /var/log/journal
reboot # give it some time before power cycling, since we want the /var/log/journal dir to be synced to disk successfully ;)

Now journalctl logs are stored to /var/log/journal. So when you now reboot again, you can then call journalctl and should see all logs from the previous shutdown sequence, at least until systemd-journald itself is shut down. This includes also kernel logs, in case they occur.

[yandritos]

journalctl

Hi Michalng, I don't got if you want the traces of my old dietpi which runs well (and reboots properly), or the traces of the new armbian I installed for this testing.

I am providing the traces of the armbian one which is the one not working after the reboot command. I ignored the command "dietpi-software uninstall 103" because it is not detected.

These are the output traces after "journalctl" for the last armbian release:

Note: I enabled the traces following your instructions, I executed "reboot" I waited for 5 minutes without any signal of live and then I unpluged/pluged the system getting this output for the journal

---

give me some time, collecting all in a file, the output was truncated

[yandritos]

Hi, Please find attached the full content of the journal folder.
all_logs.txt

[MichaIng]

Ah sorry, I forgot that it's an Armbian system now. Please try to disable armbian-ramlog via armbian-config, or via:

systemctl disable armbian-ramlog
reboot

I hope we see some more logs of the shutdown sequence then.

[yandritos]

Hi, just as requested

---

Last login: Thu Apr 14 21:14:34 2022 from 192.168.1.95
root@odroidc2:~# systemctl disable armbian-ramlog
Removed /etc/systemd/system/sysinit.target.wants/armbian-ramlog.service.
root@odroidc2:~# reboot

---

Then I waited for 10 mins prior to unplug/plug and get the trace file below.

all_logs2.txt