Skip to content

Over the air reflashing process

Jump to bottom Edit New page
Atharva Nandanwar edited this page Oct 5, 2023 · 16 revisions

There may be times when you need to perform the equivalent of a re-flashing of your Jetson-based device without being able to use the normal flashing process via USB. This is possible, although there are some risks, and it requires careful setup and testing.

Possible applications:

  • You need to alter the layout of the partitions in the Jetson's eMMC storage.

  • You need to update a Jetson running software based off an older version of the L4T BSP to a newer version that requires a modified layout of the eMMC and/or SPI flash (for Jetsons that have a SPI flash boot device).

  • You just need the equivalent of a full "factory reset" that restores the device to a pristine state.

This page walks through a basic example of how to do this, using tools and scripts that you can modify/adapt as needed. The example uses a Jetson-TX2 development kit target; it has also been tested with Xavier and Nano development kits.

Overview of the process

The goal here is to perform the equivalent of a USB "tegraflash" on the running device. What that entails is: erasing/reformatting the storage devices on the Jetson module and writing the correct boot code/data, kernel, rootfs, etc. so that on reboot, the device successfully boots into the image.

To do this under Linux, we can't be running in a rootfs that is mounted in the on-module storage. If your device supports external storage that is bootable, you could use that, or you could run the process entirely from an initial RAM disk loaded with the Linux kernel. The following example uses the latter approach.

Ingredients

  • The tegra-sysinstall repo contains the scripts that execute the overall process.
  • The tegra-boot-tools repo contains the tools for writing the boot partitions.
  • Example recipes for creating the initramfs image for the TX2 running an old L4T R32.1-based build are here.
  • The new image, based on L4T R32.5.0, is built from this test distro.

Key considerations

  • The flash layout from the new image build is used to generate configuration files that the tools use for correctly re-partitioning the storage devices. To ensure that the bootloaders and Linux agree on the eMMC partition layout, the primary GPT must be at least 16,896 bytes (33 512-byte sectors). (This is the case with the stock flash layouts for all recent releases of L4T.)

  • The partition_table file generated by the sysinstall-partition-layout recipe from the R32.5.0-based build must be copied into the metadata for the warrior/R32.1-based build, since that file will be part of the warrior-based initramfs.

  • The tegra-bootloader-update tool uses a BUP payload as the source of the contents for all of the boot partitions. The stock L4T BUP payload generator does not include all of the boot partition contents. Recent commits into meta-tegra include patches for the generator to include the missing pieces for TX2 and Xavier platforms. Update 10 Jun 2021: The additions to the BUP payload turned out to be incompatible with the stock L4T nv_upgrade_engine bootloader update program on the TX2 and were reworked here to create an alternate payload that contains the full complement of boot partitions for TX2-based platforms.

  • The tegra-sysinstall script expects the new rootfs image to be in tarball form, and does not perform any authentication or sanity checking on the image, so it is only usable for development purposes and should not be used in production.

Build process

  • The R32.5.0-based build includes tegra-bup-payload, which installs a BUP payload in /opt/ota_package and pulls in the bootloader update tool. The demo-image-egl image was used for this example. Note that it has IMAGE_FSTYPES set to include building a tar.gz tarball for the rootfs.

  • The sysinstall-upgrader-initramfs recipe in the warrior/R32.1-based tree builds a BUP payload containing the kernel and initrd suitable for installing with nv_update_engine on a system running an R32.1-based image. (Note that for platforms using U-Boot, installing the initrd would require a different process.)

  • The core-image-base image from the warrior/R32.1 tree was used as the starting point for the example.

Process steps

  1. Start by flashing the R32.5.0-based image directly on the TX2. Use sgdisk /dev/mmcblk0 --print to display the partition table, and save that output so you can compare the results against the partition table created later during the installation process.

  2. Boot the core-image-base image from the warrior/R32.1-based distro on the TX2.

  3. Because the filesystem size is not expanded out to the full APP partition size in this build, use mkfs.ext4 to format the UDA partition, and mount that at /mnt.

  4. rmdir /opt/ota_package, then ln -sn /mnt /opt/ota_package to provide space for the BUP payload.

  5. Use wget to download the sysinstall-upgrader-initramfs-jetson-tx2.bup-payload built in the warrior/R32.1-based build tree as /opt/ota_package/bl_update_payload.

  6. Use nv_update_engine --enable-ab, then nv_update_engine --install no-reboot to install the BUP payload. If successful, reboot.

  7. The kernel command line in the initramfs image doesn't have console= set, so be patient while the image loads (takes about a minute or so) - there is no kernel output during the boot.

  8. mkdir /var/extra, as this directory is needed as a mount point during the installation.

  9. mkdir /installer and use wget or curl to download the demo-image-egl-jetson-tx2-devkit.tar.gz tarball from the R32.5.0-based build, naming it /installer/image.tar.gz.

  10. tegra-sysinstall to start the installation process. After it reformats the eMMC, the script will display the new partition table. Verify that the partition start and end sectors match the ones displayed in step 1 (after flashing the R32.5.0 image directly). If there is a mismatch, the device will probably not boot properly.

This section could be customized for a specific delivery mechanism. For instance, instead of using wget to download the BUP payload, the package could be delivered through your preferred update mechanism. If using an A/B update scheme like the one used for tegrademo-mender it should be possible to use the filesystem in the new boot partition to host the BUP payload and image content.

Installation steps

These are the steps performed by tegra-sysinstall:

  1. The sgdisk command (from the gptfdisk package) is used to zap the GPT partition table and create all of the partitions on the eMMC, based on the configuration file at /usr/share/tegra-sysinstall/partition_table.

  2. The APP, APP_b, DATA, LOGS, and EXTRA partitions are formatted using mkfs.ext4.

  3. The EXTRA partition is mounted at /var/extra for use as temporary storage.

  4. The rootfs tarball is unpacked into the APP partition, then into the APP_b partition.

  5. The boot partitions are initialized by chrooting into the just-installed APP partition to run tegra-bootloader-update --initialize using the BUP payload and /usr/share/tegra-boot-tools/boot-partitions.conf configuration file from the just-installed rootfs.

Once the above steps are complete, the device can be rebooted, and should boot into the 32.5.0-based image.

(Please note that the tegra-sysinstall scripts were developed to test support for secure boot combined with LUKS encrypted filesystems and programming a unique machine ID in the odm_reserved fuses, so there are several functions in the scripts that can be ignored/skipped for testing the installation process.)

Things to watch out for

  • If the initramfs with the installation tools is too large for cboot to handle properly (it has some compiled-in limits on the amount of memory it can reserve for the initial RAM disk), you'll see data abort errors on the serial console.

  • If the BUP payload is missing any of the boot-related contents, the device will fail to boot when rebooting after the installation process is complete - one of the early-stage bootloaders will report errors on the serial console, and the device should go into USB recovery mode.

  • The above can also happen if there is a mismatch in the starting offsets and/or sizes of any of the boot partitions in the eMMC and the expected offsets that got built into the boot control tables for the bootloader during BUP generation. It's important that the /usr/share/tegra-sysinstall/partition_table configuration file in the initramfs gets correctly generated from the same flash layout XML file that you are using for the image you are upgrading to.

  • Any power interruption or other event that causes the device to reset or reboot, or otherwise interrupt the reflashing process, will render the device unbootable. Since the process can take several minutes (depending on the specific hardware, size of the image being installed, etc.), use of this process should be managed carefully.

  • Full BUP support in meta-tegra, covering multiple module revisions in a single payload, was added with the update to L4T R32.3.1. If you are currently running builds based off an older version of L4T, you may run into boot issues after installing the upgrader BUP payload on some TX2 modules. Adjusting the TEGRA_FAB setting in your build configuration to match the actual FAB revision of the module(s) you're using should help with this.

Video walkthrough

See the OE4T May 2021 meeting video and notes for initial discussion and walkthrough of the content discussed here.

Testing

Add a custom footer
Add a custom sidebar
Clone this wiki locally