Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NVIDIA Xavier NX Patch Update Request #8079

Closed
xxEoD2242 opened this issue Dec 28, 2020 · 9 comments
Closed

NVIDIA Xavier NX Patch Update Request #8079

xxEoD2242 opened this issue Dec 28, 2020 · 9 comments

Comments

@xxEoD2242
Copy link

Required Info
Camera Model D455
Firmware Version 12.10.0
Operating System & Version Ubuntu 18.04
Kernel Version (Linux Only) 4.9 Tegra
Platform Nvidia Xavier NX
SDK Version 2.41.0
Language N/A
Segment Robot

Issue Description

So, I've been trying to get the D455 to work on a Jetson Xavier NX for the past few weeks. I had some success but, in general, it was very inconsistent.

I eventually figured out that the Tegra kernel is the issue (and I know Intel doesn't officially support it). So, I attempted to use the L4T kernel patches to patch UVC and v4l. However, when I went to apply the patches (which are from 2018), the first three patches all failed at some part.

This is due, as I found out, since the patches are not even close to the latest edition of the 4.9 kernel. My method to solve this was by manually editing the kernel-4.9 Tegra copy. I then commented out the commands for applying the patches and then completed the kernel patch process.

What I'd like to request is if someone could go and update the LRS patches so that they are consistent with the latest Tegra kernel. Some of the patches that failed were way off (by 350 lines) but some of the patches for UVC were already in there (like adding support for HDI IMU).

I can provide more detail on what is not up to date and what is. I also wrote down what I changed manually and can provide that as well.

Resolution

The results of this were that (only upon initially starting depth images), I get a few Resource Not Available Messages and then the camera works perfectly, including the IMU-fused data. What appears (to me and what the heck to do I know) is that the initial UVC communication messages aren't synced properly with the RealSense SDK, causing a brief issue.

I tested several of the built-in examples and everything seems to be working fine. I'm pretty sure, however, that additional UVC patches are most likely needed.

@MartyG-RealSense
Copy link
Collaborator

MartyG-RealSense commented Dec 28, 2020

Hi @xxEoD2242 Are you getting Hunk errors when you build for Xavier? If so, having Jetpack 4.4 (L4T-32.4.3) installed may help if you are not using that version already, as described in the recent case in the link below.

#7987

@xxEoD2242
Copy link
Author

xxEoD2242 commented Dec 28, 2020

Hi @xxEoD2242 Are you getting Hunk errors when you build for Xavier? If so, having Jetpack 4.4 (L4T-32.4.3) installed may help if you are not using that version already, as described in the recent case in the link below.

#7987

Hi @MartyG-RealSense, I am using Jetpack 4.4.1, which is the latest version NVIDIA recommends using for the Xavier NX. I would assume the issue is related to the updated Jetpack then?

Yes, for the first three patches, I got Hunk errors since the files were vastly different and the patch sub-routine can only do so much about files being vastly different then expected. However, when manually updating UVC, I noticed that some of the functionality that was being added to UVC already exists in the newest Tegra kernel. This would lead me to believe that some portions of the updates were taken care of by NVIDIA since 2018 when the patches were written.

@MartyG-RealSense
Copy link
Collaborator

In the linked-to case, Jetpack 4.4 solved one RealSense user's build problems, but not a 4.4.1 user's. Would it be possible to step back one version to Jetpack 4.4 and see whether it makes a difference?

https://developer.nvidia.com/jetpack-sdk-44-archive

@xxEoD2242
Copy link
Author

xxEoD2242 commented Dec 28, 2020

Well, I got mine to work with the newest of everything so I'll probably keep that version. However, I have another development board I can use to test our Jetpack 4.4 and confirm that works for you.

I just wanted to make ya'll aware of the patch issues. I'll also take a look at UVC as well. I'd like to understand how UVC is encoding the messages, because I have a theory that the output message during startup for the firmware is misaligned with the Tegra UVC drivers.

@MartyG-RealSense Is the firmware for the RealSense camera open source?

@MartyG-RealSense
Copy link
Collaborator

I have passed your kindly provided patching feedback in this case to RealSense team members involved in JetPack and the patching script.

The firmware is not open-source.

@MartyG-RealSense
Copy link
Collaborator

Hi again @xxEoD2242 I have been informed by Intel that the L4T patch currently supports up to JetPack 4.4 (L4T-32.4.3) at the time of writing this.

@xxEoD2242
Copy link
Author

Hey @MartyG-RealSense, sounds good. I may write my own kernel patches for 4.4.1 and post them in Jetson Hacks. I'll link here when I finish them if the team is interested. Thanks for the quick turnaround!

@MartyG-RealSense
Copy link
Collaborator

You are very welcome. You are of course free to pursue your own solution, or wait for an official patch script update from Intel for 4.4.1 support. I cannot provide a time estimate of when an official solution will be available though. Thanks again for your excellent feedback!

@dlee640
Copy link

dlee640 commented Jan 27, 2021

@xxEoD2242 Hello. I am using the 4.4.1 Jetpack on Jetson Xavier. The problem is that Xavier is embedded within the robot so I do not have access to SD card slot, and it seems like I need to reinstall the jetpack 4.4 since 4.4.1 does not work. Is Intel working on getting this to work with 4.4.1? or have you been able to patch it correctly? If you were able to work this around, would you be kind enough to share your link in Jetson Hacks? I could really use some help here.. Thank you very much!

My error:

Apply Librealsense Kernel Patches
patching file drivers/media/usb/uvc/Makefile
patching file drivers/media/usb/uvc/uvc_driver.c
Hunk #1 succeeded at 764 with fuzz 2 (offset 595 lines).
patching file drivers/media/usb/uvc/uvcvideo.h
Hunk #1 succeeded at 392 with fuzz 2 (offset 254 lines).
patching file drivers/media/v4l2-core/v4l2-ioctl.c
Hunk #1 FAILED at 1262.
1 out of 1 hunk FAILED -- saving rejects to file drivers/media/v4l2-core/v4l2-ioctl.c.rej
patching file include/uapi/linux/videodev2.h
Hunk #1 FAILED at 641.
1 out of 1 hunk FAILED -- saving rejects to file include/uapi/linux/videodev2.h.rej

My jetpack version:
unitree@nx:~/librealsense$ sudo apt-cache show nvidia-jetpack
Package: nvidia-jetpack
Version: 4.4.1-b50
Architecture: arm64
Maintainer: NVIDIA Corporation
Installed-Size: 194
Depends: nvidia-cuda (= 4.4.1-b50), nvidia-opencv (= 4.4.1-b50), nvidia-cudnn8 (= 4.4.1-b50), nvidia-tensorrt (= 4.4.1-b50), nvidia-visionworks (= 4.4.1-b50), nvidia-container (= 4.4.1-b50), nvidia-vpi (= 4.4.1-b50), nvidia-l4t-jetson-multimedia-api (>> 32.4-0), nvidia-l4t-jetson-multimedia-api (<< 32.5-0)
Homepage: http://developer.nvidia.com/jetson
Priority: standard
Section: metapackages
Filename: pool/main/n/nvidia-jetpack/nvidia-jetpack_4.4.1-b50_arm64.deb
Size: 29412
SHA256: ec502e1e3672c059d8dd49e5673c5b2d8c606584d4173ee514bbc4376547a171
SHA1: 75a405f1ad533bfcd04280d1f9b237b880c39be5
MD5sum: 1267b31d8b8419d9847b0ec4961b15a4
Description: NVIDIA Jetpack Meta Package
Description-md5: ad1462289bdbc54909ae109d1d32c0a8

Package: nvidia-jetpack
Version: 4.4-b186
Architecture: arm64
Maintainer: NVIDIA Corporation
Installed-Size: 194
Depends: nvidia-cuda (= 4.4-b186), nvidia-opencv (= 4.4-b186), nvidia-cudnn8 (= 4.4-b186), nvidia-tensorrt (= 4.4-b186), nvidia-visionworks (= 4.4-b186), nvidia-container (= 4.4-b186), nvidia-vpi (= 4.4-b186), nvidia-l4t-jetson-multimedia-api (>> 32.4-0), nvidia-l4t-jetson-multimedia-api (<< 32.5-0)
Homepage: http://developer.nvidia.com/jetson
Priority: standard
Section: metapackages
Filename: pool/main/n/nvidia-jetpack/nvidia-jetpack_4.4-b186_arm64.deb
Size: 29376
SHA256: 7f90f2def712d993bd84855355aec47a1b63ae54edefcbf4c963dbe0cb6fbf92
SHA1: bd1f0b05adbdf8f0df4dce4f655bb9615172594f
MD5sum: f9d2f9a9dfee794ef361b1bd1ec38130
Description: NVIDIA Jetpack Meta Package
Description-md5: ad1462289bdbc54909ae109d1d32c0a8

Package: nvidia-jetpack
Version: 4.4-b144
Architecture: arm64
Maintainer: NVIDIA Corporation
Installed-Size: 195
Depends: nvidia-container-csv-cuda (= 10.2.89-1), libopencv-python (= 4.1.1-2-gd5a58aa75), libvisionworks-sfm-dev (= 0.90.4.501), libvisionworks-dev (= 1.6.0.501), libnvparsers7 (= 7.1.0-1+cuda10.2), libnvinfer-plugin-dev (= 7.1.0-1+cuda10.2), libnvonnxparsers7 (= 7.1.0-1+cuda10.2), libnvinfer-samples (= 7.1.0-1+cuda10.2), libnvinfer-bin (= 7.1.0-1+cuda10.2), libvisionworks-samples (= 1.6.0.501), libvisionworks-tracking-dev (= 0.88.2.501), vpi-samples (= 0.2.0), tensorrt (= 7.1.0.16-1+cuda10.2), libopencv (= 4.1.1-2-gd5a58aa75), libnvinfer-doc (= 7.1.0-1+cuda10.2), libnvparsers-dev (= 7.1.0-1+cuda10.2), libnvidia-container0 (= 0.9.0beta.1), nvidia-container-csv-visionworks (= 1.6.0.501), cuda-toolkit-10-2 (= 10.2.89-1), graphsurgeon-tf (= 7.1.0-1+cuda10.2), libcudnn8 (= 8.0.0.145-1+cuda10.2), libopencv-samples (= 4.1.1-2-gd5a58aa75), nvidia-container-csv-cudnn (= 8.0.0.145-1+cuda10.2), python-libnvinfer-dev (= 7.1.0-1+cuda10.2), libnvinfer-plugin7 (= 7.1.0-1+cuda10.2), libvisionworks (= 1.6.0.501), libcudnn8-doc (= 8.0.0.145-1+cuda10.2), nvidia-container-toolkit (= 1.0.1-1), libnvinfer-dev (= 7.1.0-1+cuda10.2), nvidia-l4t-jetson-multimedia-api (>> 32.4-0), nvidia-l4t-jetson-multimedia-api (<< 32.5-0), libopencv-dev (= 4.1.1-2-gd5a58aa75), vpi-dev (= 0.2.0), vpi (= 0.2.0), libcudnn8-dev (= 8.0.0.145-1+cuda10.2), python3-libnvinfer (= 7.1.0-1+cuda10.2), python3-libnvinfer-dev (= 7.1.0-1+cuda10.2), opencv-licenses (= 4.1.1-2-gd5a58aa75), nvidia-container-csv-tensorrt (= 7.1.0.16-1+cuda10.2), libnvinfer7 (= 7.1.0-1+cuda10.2), libnvonnxparsers-dev (= 7.1.0-1+cuda10.2), uff-converter-tf (= 7.1.0-1+cuda10.2), nvidia-docker2 (= 2.2.0-1), libvisionworks-sfm (= 0.90.4.501), libnvidia-container-tools (= 0.9.0beta.1), nvidia-container-runtime (= 3.1.0-1), python-libnvinfer (= 7.1.0-1+cuda10.2), libvisionworks-tracking (= 0.88.2.501)
Conflicts: cuda-command-line-tools-10-0, cuda-compiler-10-0, cuda-cublas-10-0, cuda-cublas-dev-10-0, cuda-cudart-10-0, cuda-cudart-dev-10-0, cuda-cufft-10-0, cuda-cufft-dev-10-0, cuda-cuobjdump-10-0, cuda-cupti-10-0, cuda-curand-10-0, cuda-curand-dev-10-0, cuda-cusolver-10-0, cuda-cusolver-dev-10-0, cuda-cusparse-10-0, cuda-cusparse-dev-10-0, cuda-documentation-10-0, cuda-driver-dev-10-0, cuda-gdb-10-0, cuda-gpu-library-advisor-10-0, cuda-libraries-10-0, cuda-libraries-dev-10-0, cuda-license-10-0, cuda-memcheck-10-0, cuda-misc-headers-10-0, cuda-npp-10-0, cuda-npp-dev-10-0, cuda-nsight-compute-addon-l4t-10-0, cuda-nvcc-10-0, cuda-nvdisasm-10-0, cuda-nvgraph-10-0, cuda-nvgraph-dev-10-0, cuda-nvml-dev-10-0, cuda-nvprof-10-0, cuda-nvprune-10-0, cuda-nvrtc-10-0, cuda-nvrtc-dev-10-0, cuda-nvtx-10-0, cuda-samples-10-0, cuda-toolkit-10-0, cuda-tools-10-0, libcudnn7, libcudnn7-dev, libcudnn7-doc, libnvinfer-plugin6, libnvinfer6, libnvonnxparsers6, libnvparsers6
Homepage: http://developer.nvidia.com/jetson
Priority: standard
Section: metapackages
Filename: pool/main/n/nvidia-jetpack/nvidia-jetpack_4.4-b144_arm64.deb
Size: 30376
SHA256: 0556dbf2044a9d12e2c26e1a342033d4816f5d84f87dfe7a82916449249bb04b
SHA1: 38e788c5cd84b3e4bd956dcd7c9e018276e17b18
MD5sum: 0a96d0fa91591f10b7c86022d068970d
Description: NVIDIA Jetpack Meta Package
Description-md5: ad1462289bdbc54909ae109d1d32c0a8

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants