-
Notifications
You must be signed in to change notification settings - Fork 4.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
T265 stops sending data #6362
Comments
@RealSenseCustomerSupport @RealSense-Customer-Engineering @MartyG-RealSense Can you help us? Is this a known issue and if so, is there a workaround or proposed fix? Any additional information we can collect for you too look at? This problem renders the T265 unusable for our project. |
We did some more investigation and it looks like a firmware or a libusb issue. This is how the pose/gyro/accel is being received from the T265 in librealsense:
Steps 2-4 are repeated. As long as the callback function We implemented a watchdog thread which monitors the callback and if the callback is not called anymore after some time we call stop_interrupt() followed by a start_interrupt(). We were hoping this would reinitialize the callback and cause the data to show up again. But this didn't help. Only a relaunch of the ROS node helped. Unfortunately we cannot go deeper since we don't have the firmware of the T265. But our guess is that the T265 just wouldn't respond anymore. @RealSenseCustomerSupport @RealSense-Customer-Engineering @MartyG-RealSense could you please see what could cause the callback function not getting called anymore? Are there logs I can enable in the T265 firmware? |
We ran the ROS node in debug mode. These are the last few lines before T265 stops sending data:
After the last line there's no more output. |
I have also been seeing this exact issue, with the same setup. RPI4 with same OS, SDK version, with T265 and D435i. Happens with both ROS and simple C++ test scripts. Data comes in from the T265 (using Other Observations:
Personally have not had any success with This problem is currently a showstopper for me, which is unfortunate since during the windows when it works the sensor is pretty great. I might try rolling back to 2.31, but based on what I read here and the release notes that may just be trading one set of problems for another. Please help. |
I switched to the development branch (2.35). The D435i is not recognized at all, but the T265 seemed to work better, but still crashes. The D435i shows up properly as a usb device:
but the SDK refuses to detect it. The rs-enumerate-devices example does not see it, nor do tools like the fw-logger/fw-update. As a side note, both sensors have always worked when plugged into a windows computer. |
Using latest development branch at commit 2a79d31 merged by @ev-mp Compiled and ran the Attached is the log, failure after about 60 seconds of running. The message received after failure is:
Anyone have any ideas? edit: Using
Also, probably not important but when running at this debug level properly we get a flood of: Max duration is calculated as There is also this:
Pose #0 is always overdue, every single cycle, with duration being equal to the DispatchedAt time. |
Glad to hear I'm not the only one with this problem, @csr-kick. When you tried with the latest dev branch and you got the And is it still true that a |
I can replicate the failure without the D435i plugged in, but it does seem to make it happen faster when it is running. The data I am collecting now is as bare as possible, running the rs-pose example with only the T265 plugged in at all to the Rpi4. edit: When I do variations on
|
@MartyG-RealSense @dorodnic Any ideas/help on this issue? 'Host not responding' error makes me think the SDK stops processing messages or maybe requesting the message in the first place, so the sensor stops as well. |
@csr-kick I was just thinking, what if you create a new pipe after stopping the old one? This should release any resource that the old pipe holds:
I'm not home right now so I can't test this myself. Edit: Here's the code that worked for me
@csr-kick can you give this a try? |
Using your code snippet does recover, but I want to try it a few times with the D435i also running. However, this isnt a viable solution beyond basic prototyping.
It really seems like somewhere in the SDK it stops processing the pose data coming from the sensor. The sensor sees the messages backing up and it stops sending to avoid any issues. The bulk data transfers and time syncing continue to work, which tells me that it is specific to whatever protocol the pose message uses. We also know that none of this happens if you plug into a windows computer and run the realsense viewer app, so its probably low in the code stack. |
I agree that this is not a fix and ultimately we need something better than restarting the pipe. I think the workaround can work fairly well though if configured properly but we need to do some more testing. To your questions: Re 1: Re 2: I might be wrong on this but the way I understand is when you reset the pipe, you are only reconnecting to the T265, you're not hard resetting the camera. In order to hard reset the T265 you'd have to call hardware_reset(). That means your camera will keep the last pose even after you've reset the pipe. Re 3: Since the T265 didn't hard reset you shouldn't need to re-import the localization map. Would be nice to have someone from Realsense chime in on this to make sure I'm not mistaken. |
Thanks, forgot about the timeout. Make sure that you set it initially to a high value to allow the sensor boot up or whatever before dropping the timeout down for normal operations. Then in the catch statement put it back up. My current testing code:
My testing shows that (without loading any maps right now) localization does go back to 0,0,0. In extended testing this has thrown a segfault after quite a while, typically when coming back up. Seems like a pose message from before the pipe.stop comes back after we restart and it doesn't like that. @RealSenseSupport can we get any sort of statement about this problem? |
Still working to see if there is some way to actually make the T265 work with the RPI. I have tried setting up ubuntu 18 on the rpi, and same problem, T265 will eventually stop responding, and it happens in a matter of seconds with the D435 sending depth at 640x360x30fps. @MartyG-RealSense please can we get some sort of response from intel on this issue. |
Just wanted to put out there that I'm having the exact same problem and am also restarting the pipeline on failure. I'm on v2.34.1. Would love a fix to this @MartyG-RealSense |
Same problem here with the SDK v2.35.2 |
Hi all, Thanks. |
Thanks for highlighting this. At this time we have moved our focus to our next generation of products and consequentially will not be addressing this T265 issue. |
@RealSenseSupport are you dropping support for the T265? I don't see any mention on https://www.intelrealsense.com/tracking-camera-t265 |
Same issue here, I did not find an information about T265 not being supported. |
I am now able to provide information about the future of RealSense, as an 'End of Life' notice has been released by Intel. Although some RealSense models are being discontinued, the 400 Series stereo depth cameras are continuing. The official PDF of the End of Life notice explaining the changes is attached below. |
Thank you for this update @MartyG-RealSense. As the T265 will be de-supported, can at least the source code of the firmware be made public? Even if Intel isn't interested in fixing this bug, many other in the community will be eager and capable of tackling this problem if we just had access to the source code. |
Hi @dschnabel Whilst the RealSense SDK is open-source and developers are free to 'fork' it to create their own custom versions that they host on their own GitHub, Intel do not open-source RealSense's firmwares or camera algorithms. |
i think you just highlighted the problem... maybe i can ask 'do not', to 'to the reason why'? |
@nixinator A reason given in the past has been that such resources are Intel's intellectual property. |
root cause analysis...complete! |
If you can't release the source code, can you solve this issue before dropping support? Perhaps you could release part of the source code? |
@zhouzhiwen2000 Development work for the T265 has ended so no further software or firmware updates can be provided for it. |
This is why no one will ever trust the depth sensors that are still around. Companies cannot base their products on a device where the creator has no 'anchor' product or vested interest in it's success and support. If you have an issue and wake up the next day to find that the supplier has decided to simply walk away rather than do the right thing for their customers, your existence is at risk while theirs is not. |
And in the case of the problem unplug then plug after first boot up, why don't T265 try turn off then on itself when launch the node? For example, I saw RPLidar a1 keeps rotate and when I launch the node RPLidar a1 on ROS, it stops rotate for 1~2 seconds and rotate again. Even that Lidar connected to the usb hub, still works. |
After plugging in and out about 200 times on the Rpi, I found out that you can use Uhubctl to automatically power cycle all the USB ports after boot, so that the t265 is recognized all the time. |
nice work and power fluctuation debugging, nice. can i ask what power meter you are using in your screenshot, probably a essential item for robot builders. |
@nixinator My bad, it's a photo not a screenshot😂 |
@HanDaSeul When I did my testing the RPI4 (4GB) was sitting on my desk with just the T265 plugged in. It happens more frequently when you stress the CPU, and if you stress the USB by also taking in the video streams. When I used a powered USB hub with the D453i it also crashed more frequently, but as you pointed out not all usb hubs can provide the right power. I'll do a quick check with a custom USB hub I have somewhere around here that has its own 5v/5A regulator on it. Can you tell me what USB power analyzer you have? Looks pretty nice. As we talked earlier in this thread, I wrapped the driver in code that tracks the last pose, restarts the T265 pose stream, then applies the old pose as an offset to the now zero'd poses. Some other trickery with matching depths or lidar scans will relocalize any drift pretty well. |
@csr-kick If i'm right, USB power analyzer = power meter = 'j7-c'. As you mentioned about stress, I tried overclocking little bit higher and making sure keep cool(didn't exceed 50 celsius degree), but nothing changed. Through the whole test myself, conclusion is 'as long as you are using RPi, don't try streaming in ROS'. But just realsense-viewer, you can try streaming and there's no problem with connection. I still can't understand why. Only I can make it clear is there's difference of using resources like cpu, power whatever. I don't have D453i, or kinda depth. That's why I'm trying make wheel odometry model for my project. |
I don't think it's a power delivery issue. At least not from my tests: The Pi is sucking back around 0.8-0.9A with the T265 running. The USB power bank I'm using is capable of delivering 2.5A, so there's no bottleneck there. |
@Construkted-Reality I tried several times after I applied my solution, there's no losing connection. Sometimes, having without usb tester is helpful. In my experience, which is not exactly clear tho, powerbank is less stable than power adapter. RPi 4 8GB is recommended 3A. for input. I tested with 3A adapter for RPi, 2A for powered usb hub. I'm using wireless usb adapter for sending pose data and it takes 0.45A, 5V, which means I turned off RPi's wifi chip. And I disabled fisheye camera streaming, too(maybe you already tried). |
@HanDaSeul |
@Construkted-Reality Then, probably only I can say is use wireless usb wifi adapter with powered usb hub. Don't let other usb devices take power from RPi. Let RPi take less power as you can like setting RPi's wifi chip off. And just in case keep T265 cool. Wait...it looks like the problem is not only the power, but also RPi's wifi chip performance...I didn't test using RPi's wifi chip. |
I don't think cooing the T265 has anything to do with it since I've had the T265 work perfectly on a Pi3, and it never disconnects (without any cooling). |
Try putting more pressure on the RPI USB port while testing, eg. copying a large file from an SSD. It seems that the problem occurs more often when the bus is busy. The SSD can be powered externally to avoid extra power consumption. |
I tested using T265 and a USB camera, with the USB cam externally powered, and the problem is still around. |
For those using T265, the ASUS tinker board seems to be a good choice. It's tested and no such problem is observed. Its price is also similar to RPI4. |
If you guys seach here about T265, that camera doesnt work properly in Raspberry 4 / Nvidia Jetson Nano, lots of people complaining about this (inluding myself), and Intel will not correct this (just shutdown after a while, and have to take out the cable and reconnect again). But works in Raspberry 3(not tested by myself), some people confirmed this. |
no, I 'm facing the similar problem in raspberry pi 3b+ |
did u mean that using uhubctl to manage power so that t265 will be connected all the time without odom losing? |
The odom dropping out is one issue. The RS not being recognized after
reboot (if I remember correctly) is another issue, one that is fixed using
ububctl.
…On Wed, 15 May 2024 at 04:45, Truong Giang Nguyen ***@***.***> wrote:
After plugging in and out about 200 times on the Rpi, I found out that you
can use Uhubctl to automatically power cycle all the USB ports after boot,
so that the t265 is recognized all the time.
did u mean that using uhubctl to manage power so that t265 will be
connected all the time without odom losing?
—
Reply to this email directly, view it on GitHub
<#6362 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AFAMSECFYN4JSFFBFINZLB3ZCLD3DAVCNFSM4M3CLYZ2U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TEMJRGE2DONZZGI3A>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
--
*Ing. Teymur Azayev PhD*
*+420 725767494*
|
0.2.0.926
, D435:05.12.03.00
Issue Description
After upgrading from SDK 2.31.0 to 2.33.1 my T265 stops sending pose data when running for some time (1-5 min). This happens if the T265 is run together with the D435 and also if I run the T265 alone (less frequently).
This wasn't a problem in 2.31.0 but I was experiencing crashes in 2.31.0 so I decided to upgrade to the latest stable version (2.33.1).
In 2.33.1 I see no more crashes but I seem to be experiencing a very similar problem as described in #5509 (comment).
I saw some discussion on missing serial numbers but in my case both cameras seem to get detected properly (see my ROS startup logs for T265 and D435).
When I monitor the odom topic from the T265, after a while I don't get any more pose (odom) data:
For completeness here are my two ROS launch files:
Is there any fix for this?
The text was updated successfully, but these errors were encountered: