Update NVIDIA bootloader state on every boot

[madisongh]

No description provided.

[jajoosiddhant]

Forgetting to mender commit scenario:
Mender updating to RootfsPartB (uboot):

Boot number	Priority slot 0, retry_count, boot_successful	Priority slot 1, retry_count, boot_successful	Nvbootctrl get-current-slot	Fw_printenv -n mender_boot_part
After Mender update and before reboot	14,7,1	15,7,0	0	33
0	14,7,1	15,7,1	1	33
Forget to Mender commit and reboot
1 (Mismatch here)	14,7,1	15,7,1	1	1
2 (Mismatch here) - if we never mender commit we would always be in a misaligned state for future updates and reboots	14,7,1	15,7,1	1	1
Mender commit here in boot 2 will give an error and rollback using ArtifactRollback_Leave_80_bl-rollback
Info after error in mender commit and before reboot (This means that the boot partition has been rolled back to previous one i.e slot 0)	15,7,0	14,7,1	1	1
3 (Now the partitions are aligned)	14,7,1	15,7,1	0	1
4	15,7,1	14,7,1	0	1
Any subsequent reboots should be aligned

The PR works when the user mender commits after an update but we are stuck with misaligned partitions if the user forgets to mender commit and reboots. Please check the table above for details which were produced using this PR.
This can be taken care with an auto commit feature to commit whenever standalone mender update is detected as per logic relevant to auto mender commit here
OR
Detect whether the user has forgot to do a mender -commit and execute the ArtifactRollback_Leave_80_bl-rollback script which realigns the partitions in case of mismatch.
@madisongh What are you views on this?

[jajoosiddhant]

Just tested mender update and rollback on cboot as well.
It seems that mender -commit is redundant in case of cboot. The update persists irrespective of doing a mender -commit after mender updates.

However, cboot does not rollback to the other partition on executing mender -rollback

root@jetson-tx2:/data# ./slot_info.sh
Dump Slot Info:
magic:0x43424e00,             version: 3             features: 3             num_slots: 2
slot: 0,             priority: 14,             suffix: _a,             retry_count: 7,             boot_successful: 1
slot: 1,             priority: 15,             suffix: _b,             retry_count: 7,             boot_successful: 1
Current Slot: 1
Mender Boot Part: 1
/dev/mmcblk0p33          10.7G    446.8M      9.7G   4% /
/dev/mmcblk0p34           4.0G    465.0M      3.3G  12% /data

root@jetson-tx2:/data# mender -rollback
INFO[0000] Loaded configuration file: /var/lib/mender/mender.conf
INFO[0000] Loaded configuration file: /etc/mender/mender.conf
INFO[0000] Mender running on partition: /dev/mmcblk0p33
INFO[0000] Update Module path "/usr/share/mender/modules/v3" could not be opened (open /usr/share/mender/modules/v3: no such file or directory). Update modules will not be available
Rolling back Artifact...
INFO[0000] Setting partition for rollback: 1

root@jetson-tx2:/data# reboot
Connection to 10.1.10.214 closed by remote host.
Connection to 10.1.10.214 closed.

root@jetson-tx2:/data# ./slot_info.sh
Dump Slot Info:
magic:0x43424e00,             version: 3             features: 3             num_slots: 2
slot: 0,             priority: 14,             suffix: _a,             retry_count: 7,             boot_successful: 1
slot: 1,             priority: 15,             suffix: _b,             retry_count: 7,             boot_successful: 1
Current Slot: 1
Mender Boot Part: 1
/dev/mmcblk0p33          10.7G    446.8M      9.7G   4% /
/dev/mmcblk0p34           4.0G    465.0M      3.3G  12% /data

Maybe modifying redundant-boot-rollback-script-uboot to deliberately switch boot partitions for rollback in case of cboot might fix it?

[madisongh]

However, cboot does not rollback to the other partition on executing mender -rollback

What happens in a rollback situation? Let's say you're on rootfs A to start with:

1. You mender install the new image, which gets loaded into rootfs B
2. On reboot, the device tries booting the new rootfs in slot B, but fails to boot (however many times)
3. The bootloader switches back to rootfs A after the failures
4. mender rollback is run to tell Mender about the rollback as part of the successful boot (or the mender-client service detects the failure and rolls itself back).

Since the device is already booting from slot A again, there's nothing for us to do as far as cboot is concerned, correct? It has already rolled back to the good slot.

[madisongh]

@madisongh What are you views on this?

My original view on this was that manual installs are not the normal production use case, so if people shoot themselves in the foot, so be it. I changed my mind on this after having to clean up after other developers and QA engineers that ended up with mismatched devices who would either forget to mender commit after reboot, or thought that they could mender commit before rebooting as a short cut.

What I did at work to deal with this (until I switched away from the combined cboot + U-Boot setup) was to have an ArtifactInstall_Enter_10_bootloader-sanity-check state script that would check to see if the boot slot and U-Boot variables were aligned, and if not, issue a warning message and create a sentinel file to tell the ArtifactInstall_Leave_80_bl-update state script to skip the update of the NVIDIA bootloaders and the boot slot switch - so that on reboot, the two would be back in sync again.

In most cases the bootloader update payload wasn't critical to the install, so this generally worked OK, and saved me a bunch of grunt work cleaning up after folks. There's some risk that something (like a device tree change) wouldn't get picked up due to skipping the bootloader update payload step, but for my situation, at least, that was much less likely to be a problem.

[dwalkes]

Thanks for the responses @madisongh

2. On reboot, the device tries booting the new rootfs in slot B, but fails to boot (however many times)
3. The bootloader switches back to rootfs A after the failures

Since the device is already booting from slot A again, there's nothing for us to do as far as cboot is concerned, correct? It has already rolled back to the good slot.

Agree this scenario is handled. What about this one though?

1. User initiates update from hosted mender on rootfs A, installs into rootfs B.
2. On reboot, the device boots successfuly in rootfs B, but can no longer contact the mender server (network driver, proxy handling, etc is messed up in the new image).
3. No mender commit is issued by the mender server (since it can't be reached) and mender rollback occurs.
4. The device reboots again.

In the scenario above I think with these changes, if we don't support rollback we'll still be on rootfs B. If we supported mender rollback we'd be on rootfs A and our previous configuration which was able to successfully access the hosted mender server. Do I understand this correctly?

[dwalkes]

My original view on this was that manual installs are not the normal production use case, so if people shoot themselves in the foot, so be it. I changed my mind on this after having to clean up after other developers and QA engineers that ended up with mismatched devices who would either forget to mender commit after reboot, or thought that they could mender commit before rebooting as a short cut.

My view on this is that it's too easy to mess up the u-boot configuration as it currently is and we should either have changes like those proposed in sighthoundinc#1 which automate the commit step (which so far haven't been resoundingly approved at https://hub.mender.io/t/auto-commit-for-standalone-mender-updates/2791/3 ) or we need documentation warning against using this configuration for anyone starting out with TX2, possibly with a warning we add in bitbake if you build jetson-tx2 with bootloader other that cboot. If/when we support redundant boot on Nano I'd lobby hard for an auto commit step like https://hub.mender.io/t/auto-commit-for-standalone-mender-updates/2791/3 and then maybe we could remove the warning/suggestion not to use uboot on jetson-tx2. In the meantime, we've found that a successful mender update can take you from u-boot to c-boot so this is one migration path out of u-boot on jetson-tx2.

[madisongh]

Agree this scenario is handled. What about this one though?

1. User initiates update from hosted mender on rootfs A, installs into rootfs B.
2. On reboot, the device boots successfuly in rootfs B, but can no longer contact the mender server (network driver, proxy handling, etc is messed up in the new image).
3. No mender commit is issued by the mender server (since it can't be reached) and mender rollback occurs.
4. The device reboots again.

In the scenario above I think with these changes, if we don't support rollback we'll still be on rootfs B. If we supported mender rollback we'd be on rootfs A and our previous configuration which was able to successfully access the hosted mender server. Do I understand this correctly?

I don't think the state machine is dependent on contacting the mender server to complete the commit; it only reports on its current state, and retries later if the server isn't reachable. If I'm wrong about that then yes, this would be an issue. Solving it reliably could be problematic, too, since we don't have visibility into the Mender client's current state.

[dwalkes]

I don't think the state machine is dependent on contacting the mender server to complete the commit; it only reports on its current state, and retries later if the server isn't reachable

I tried to find something in mender documentation which states this definitively one way or another and couldn't find one. It also wasn't obvious to me from searching the mender client code.

I see a comment from Eystein at https://news.ycombinator.com/item?id=13745959 which states

Once the mender daemon comes up it will try to report the success of the deployment to the server. If this fails it will also roll back

Which matches what I thought happened. That said I've never specifically tested this case before, I just always assumed that was the point of mender rollback.

Solving it reliably could be problematic, too, since we don't have visibility into the Mender client's current state.

Agree, and we probably don't need it. Just wanted to verify you didn't have some way this scenario would be handled.

[madisongh]

It also wasn't obvious to me from searching the mender client code.

I agree it's non-obvious (like most Go code I run across, it seems), so my read of the state machine code could be wrong, but I didn't see a hard dependency on server communication in there for the commit transition.

[dwalkes]

Looks good to me. Did you want to merge here first and then get upstream? I'd like to also include some automation scripts we are working on for testing, and test with some other machines before we upstream.

[madisongh]

Did you want to merge here first and then get upstream?

Yeah, I thought I'd merge here, using this repo as a staging area where we can collect up anything that needs to go upstream, if that's OK.

[manuel-wagesreither]

Regarding

Agree this scenario is handled. What about this one though?

1. User initiates update from hosted mender on rootfs A, installs into rootfs B.
2. On reboot, the device boots successfuly in rootfs B, but can no longer contact the mender server (network driver, proxy handling, etc is messed up in the new image).
3. No mender commit is issued by the mender server (since it can't be reached) and mender rollback occurs.
4. The device reboots again.

In the scenario above I think with these changes, if we don't support rollback we'll still be on rootfs B. If we supported mender rollback we'd be on rootfs A and our previous configuration which was able to successfully access the hosted mender server. Do I understand this correctly?

and

I see a comment from Eystein at https://news.ycombinator.com/item?id=13745959 which states

Once the mender daemon comes up it will try to report the success of the deployment to the server. If this fails it will also roll back

Our devices (which run zeus) do suffer from the following problem:

• Deployment takes place over WiFi, device reboots
• iPhone which acted as hotspot rejects attempts to reconnect. (This seems to be unique to iPhones. Android phones seem to allow reconnect.)
• System seems to run fine, but mender is unable to communicate deployment success to mender server.
• User turns off device.
• User turns on device, device boots into old slot.

Perhaps a rollback-reboot would have been triggered automatically if the device would have run long enough. We are using the hosted mender solution and will update to dunfell soon. Will this PR change anything on this behaviour?

Thank you!

[dwalkes]

Hi @manuel-wagesreither

System seems to run fine, but mender is unable to communicate deployment success to mender server.
User turns off device.
User turns on device, device boots into old slot.
Will this PR change anything on this behaviour?

And based on this post you are using u-boot as your bootloader, is that correct? If so I don't think this PR is going to change anything about the behavior. Mender is still going to roll back your u-boot boot slot and you are going to be out of sync between boot slots used for rootfs (controlled by mender) and bootloader slot controlled by NVIDIA.

I suggest migrating to cboot. With cboot as your bootloader this PR is going to avoid this scenario by only rolling back to the old boot slot if you boot 7 times without getting as far as the update-nvbootctrl.service started here and won't verify connection to the mender deployment server. You should be able to mender update from u-boot to cboot (although I haven't verified myself on Zeus).

[manuel-wagesreither]

Hi @dwalkes

thanks for your reply! Yes, we are using u-boot right now. We will look into migrating to a cboot-only solution, but I'm not sure if we want to risk such a major change now. Right now we just disallow update in case nvbootctrl and mender boot slots are out of sync. That was meant as a provisional quickfix, but I can already see it lasting longer than expected.

I remember a comment from @madisongh (at the Yocto Project Summit, I think) that one has to take it upon him/herself to propagate the systemd-machineid from one slot to the other with a cboot-only solution. He said it can be made to work, though. But that's something we will definitely have to look into before doing such a switch.

[madisongh]

I remember a comment from @madisongh (at the Yocto Project Summit, I think) that one has to take it upon him/herself to propagate the systemd-machineid from one slot to the other with a cboot-only solution.

That's right. One way I've done this is to add a state script to copy /etc/machine-id into the new rootfs during an update. That approach would probably affect delta updates, though (which I've never used). Another is to store the machine ID separately (I've used some of the RESERVED_ODM e-fuses for this) and either patch cboot to add it to the kernel command line, or pass it to systemd via the command line via a wrapper script.

[dwalkes]

Shame on me but I missed this in the yocto project talk, couldn't find it either in a quick review of the presentation. This seems like something we need to at minimum document but ideally have some way to fix in the community layer. I suppose a symlink to /data doesn't work for this?

I don't think I understand why this is only important for cboot, it seems it would be related to uboot builds as well.

[madisongh]

Shame on me but I missed this in the yocto project talk,

IIRC, I mentioned it in passing either as a side comment or in response to a question.

have some way to fix in the community layer

I just checked, and the copy-after-install mechanism I mentioned is implemented in the state scripts in the community later, if you have the mender-persist-systemd-machine-id feature enabled.

I suppose a symlink to /data doesn't work for this?

I don't think systemd would handle it being a symlink, but maybe a bind mount would work, if you mount /data in the initramfs before systemd starts. That doesn't work so well if you're using systemd in the initramfs, too, though.

it would be related to uboot builds as well

With U-boot, the machine ID is persisted in a U-Boot variable and added to the kernel command line at boot.

[dwalkes]

I just checked, and the copy-after-install mechanism I mentioned is implemented in the state scripts in the community later, if you have the mender-persist-systemd-machine-id feature enabled.

Great, and that should be setup by default here in either u-boot or cboot builds

I can see the feature that went into meta-mender at https://github.com/mendersoftware/meta-mender/pull/759/files but is only for u-boot, and I can see the logic here which makes this also work for cboot.

With U-boot, the machine ID is persisted in a U-Boot variable and added to the kernel command line at boot.

Got it, you'd think I'd remember that from mendersoftware@f6ced69 or OE4T/meta-tegra#200

That approach would probably affect delta updates, though (which I've never used).

This supposedly worked on warrior with mendersoftware#165 although I haven't tested it either. Maybe this is one reason it's not working since then.

[dwalkes]

@madisongh what are your thoughts about either upstreaming this or getting it used for https://github.com/OE4T/tegra-demo-distro/blob/dunfell-l4t-r32.4.3/.gitmodules#L19 in the short term? I know you have other changes in the works for nvidia boot tools, so the scope could ultimately be bigger than just this change, but I think this change is important for test automation. Also I'm thinking the nvidia boot tools change you have in the works is probably not destined for dunfell.

I've been able to somehow create boot slot mismatches with latest cboot testing I did today on the latest tegra-demo-distro dunfell + our custom changes so will need to go back over this during the coming week, starting with tegra-demo-distro, and I'm wondering what direction I should take. I'm thinking about starting with tegra-demo-distro + this patch because I think missing this patch will result in race conditions on boot that could cause failover in test automation.

[madisongh]

@dwalkes If you think they're good to go upstream, I can open a PR with these changes. Without something like these, you will run into the problem of the per-boot slot priority decrements triggering a slot switch after a couple of reboots on either TX2 or Xavier.

I also have patches on this branch that I think improve the cboot+U-boot case for the TX2, but they could use more thorough testing.

[madisongh]

PR: mendersoftware#206

[dwalkes]

I've been able to somehow create boot slot mismatches with latest cboot testing I did today on the latest tegra-demo-distro dunfell + our custom changes so will need to go back over this during the coming week, starting with tegra-demo-distro, and I'm wondering what direction I should take.

I may have confused myself somehow I'm since not able to reproduce this with the latest here on tegra-demo-distro or our custom distro. In the meantime I've added a new PR for the test scripts I'm using to validate and attempt to reproduce at mendersoftware#207. When this is approved I'll send a pr to tegra-demo-distro dunfell branch.

I also have patches on this branch that I think improve the cboot+U-boot case for the TX2, but they could use more thorough testing.

Looks interesting and helpful, but I'm also wondering if we should just make a change like the one below to default to cboot in tegrademo-mender.conf:

# Use cboot for tx2 targets by default to avoid the uboot complexity with mender boot slot matches.
# Override if you know what you are doing and/or need backwards compatibility
TEGRADEMO_MENDER_DEFAULT_BOOTLOADER := "${PREFERRED_PROVIDER_virtual/bootloader}"
PREFERRED_PROVIDER_virtual/bootloader = "${@'cboot-t18x' if 'jetson-tx2' in d.getVar('MACHINEOVERRIDES').split(':') else d.getVar('TEGRADEMO_MENDER_DEFAULT_BOOTLOADER')}"

Let me know if you have any suggestions about a better way to do this that might result in less prison time for crimes against yocto.

[madisongh]

Looks interesting and helpful, but I'm also wondering if we should just make a change like the one below to default to cboot...

On the one hand, I'd certainly recommend using cboot only on the TX2 for any product. On the other hand, I don't want to make it impossible to use U-Boot, if someone really needs to for some reason.

Let me know if you have any suggestions about a better way to do this that might result in less prison time for crimes against yocto.

😃
This should be sufficient:

PREFERRED_PROVIDER_virtual/bootloader_tegra186 = "cboot-t18x"

[dwalkes]

On the other hand, I don't want to make it impossible to use U-Boot, if someone really needs to for some reason.

Agreed, I think just making cboot the default for people who are getting started will be helpful.

This should be sufficient:

Thanks for keeping me out of yocto jail, not sure why I didn't think to try that. Will include in a PR along with meta-mender-community dunfell after mendersoftware#207

[dwalkes]

@manuel-wagesreither regarding upgrade from u-boot to c-boot on jetson-tx2, see the note from @madisongh at OE4T/tegra-demo-distro#80 (comment) about one step that's missing regarding forward propagating upgrade_available in case you ever decide to take that on.