T265 adding an auxiliary Serial & Power port for embedded systems.

[patrickpoirier51]

Allow me to make a System Modification Request here, for adding an auxiliary Serial & Power port for embedded systems.

This port would offer a direct connection for smaller systems to read the signal on a serial interface without having to load any specific driver. This port would also be used for power using a 5 Volts regulated supply.

Here is what could be the auxiliary pinout:

• Vcc
• TX
• RX
• GND

We would use the existing USB connected to a computer in order to set the speed and data stream.
As for example, in ArduPilot we are using this message to for the vision systems:

https://mavlink.io/en/messages/common.html#VISION_POSITION_ESTIMATE

Field Name | Type | Units | Description
usec | uint64_t | us |
x | float | m | Global X position
y | float | m | Global Y position
z | float | m | Global Z position
roll | float | rad | Roll angle
pitch | float | rad | Pitch angle
yaw | float | rad | Yaw angle
covariance **
reset_counter**

This way we could feed directly the Flight Controller (or any other embedded control system) and use the T265 without having to read and retransmit the signal using an onboard "Companion Computer'' that adds weight and complexity.

Here is a picture of my development vehicle using the Nvidia NANO as the "Companion Computer", and I show (orange line) how the system could be simplified using the auxiliary port to feed signal directly to Flight Computer , the cube.

[MartyG-RealSense]

Thank you very much for your detailed feature request and sharing your project! I should make clear that I am not an Intel employee, so the reply below is not representative of an official response, just a personal one. I'm sure that the Intel personnel on this forum will be happy to contribute their own feedback.

Existing models in the RealSense range do not tend to receive significant hardware changes. The most recent exception would be the adding of an IMU component to the D435 depth camera to create an additional enhanced model in the range called the D435i.

Whilst it is not impossible for the T265 to receive such an update in the future (something like a 'T265s'), it would depend on (a) whether Intel decide to release another T265 model, and (b) whether a compelling need for a feature such as a serial port is perceived to be something that would be widely used.

Thanks again for your suggestion - it's very welcome!

[lvale]

Agree completely with @patrickpoirier51

The current "consumer friendly" format of the RealSense cameras is great for bench use, and limited use on autonomous devices where the cables and connectors, although correct from an engineering point of view, make it hard to integrate with systems that down't have those bulky USB plugs.

I do hope that the industrial version is more versatile.

edit: as an alternative, would be great to get some guidance in "hacking" the consumer version of the RealSense cameras ;)

[patrickpoirier51]

@lvale this is why I would prefer a second port - AUX- so it can be connected with less bulky type of connector like a Jst-Gh , that offers a lock mechanism and that is standard within the PX4 & ArduPilot community and most of the academic hardware.

[radfordi]

@lvale, we offer modules, with connectors that may be more appropriate for embedded systems, for our depth cameras and we are interested in feedback/interest in our offering similar higher volume modules for the T265.

@patrickpoirier51, for your application it seem that a 90deg connector with locking screws would be more appropriate than the cable we ship (not that we haven't tested or endorse this particular cable).

It looks like you are using the PIXHAWK 2 CUBE? It has a micro USB, so it seems you can do exactly what you want and connect the T265 directly to it. The firmware is open source and it looks like you can compile it yourself. I was thinking of making a minimal example of using T265 (pose only) on top of libusb (i.e. without all of librealsense) just to help bootstrap this kind of application.

Besides VISION_POSITION_ESTIMATE, are you using VISION_SPEED_ESTIMATE? I was hoping our pose velocity estimate would be useful in your use case. Is outputting HIL_STATE_QUATERNION or CONTROL_SYSTEM_STATE supported? They use quaternions instead of roll/pitch/yaw and includes rotational and linear velocity.

[patrickpoirier51]

@radfordi Thanks for your guidance.
We are running ArduPilot http://ardupilot.org/dev/index.html on the CUBE.

The USB port is could be used but I would require some work as there is not Host side implementation of the OTG at the moment. A method for supply power to T265 is necessary, so we could build a Power Injector cable assembly. Please note that all the above are not necessary with an auxiliary port on the T265.

Nonetheless, your idea of making a minimal example of using T265 on top of libusb (i.e. without all of librealsense) seems quite appealing as we could reduce the footprint (and associated weight) by using a much more compact Companion Computer like the RPI Zero.

Please let us know when your example is available so we can test and and ultimately provide the community with an image than can be flashed on a SD.

Here is an example of a RPI-Zero and a T265 , just need the USB-Injector to make a ready to fly VIO system ;-)

[patrickpoirier51]

@radfordi Jim, just an update here, I finally decided to start the RPI Zero code building marathon and after 3 days, it is running ..... but with some limitations.

From our python script we can only get 10 Hz pose and the confidence level is at medium (cannot go higher even if we move camera as with other tested systems), making this configuration quite marginal for a UAV, it would probably work ok on a ROVER.

We are using this script:
https://github.com/hoangthien94/vision_to_mavros/blob/master/scripts/t265_to_mavlink.py

Here are the pictures of the setup , there is a 5Volt 3Amps UBEC under the pi for supllying both the Pi and the T265:

It is installed in a 330mm size vehicle with PixRacer Flight Controler , running ArduPilot 3.7

[patrickpoirier51]

Here are some notes:
https://discuss.ardupilot.org/t/integration-of-ardupilot-and-vio-tracking-camera-part-1-getting-started-with-the-intel-realsense-t265-on-rasberry-pi-3b/43162

Increase swap size by editing variable CONF_SWAPSIZE=2048
sudo nano /etc/dphys-swapfile

Install dependencies
sudo apt install git libssl-dev libusb-1.0-0-dev pkg-config -y
sudo apt install cmake python3-dev raspberrypi-kernel-headers -y
git clone https://github.com/IntelRealSense/librealsense.git

Set vfp
#3490
the compiler flag -mfpu=neon will not work for the pi zero as it uses a ARMv6 processor
instead of the ARMv7 processor in the pi 3. You have to rebuild with the flag set to -mfpu=vfp
in file CMake/unix_config.cmake

Set atomic
#4375

cmake -D CMAKE_BUILD_TYPE="Release"
-D FORCE_LIBUVC=ON
-D BUILD_PYTHON_BINDINGS=ON
-D BUILD_EXAMPLES=ON
-D BUILD_GRAPHICAL_EXAMPLES=OFF ..

sudo make install
sudo ldconfig
sudo reboot

Test
pip3 install transformations
python3 t265_example.py

---Get T265 ready------
pip3 install dronekit
pip3 install apscheduler
sudo pip3 install pyserial

[MartyG-RealSense]

@patrickpoirier51 That's an awesome custom setup - thanks so much for sharing your work in such detail for other members of the community to benefit from! :)

[radfordi]

From our python script we can only get 10 Hz pose and the confidence level is at medium (cannot go higher even if we move camera as with other tested systems),

@patrickpoirier51, I can see why you wouldn't get 200Hz, if say writing to vehicle is slow, but I can't see why you wouldn't get high confidence as this is entirely within the firmware. To investigate the former, would you add timing code to your main loop to see where and how much time is spent on each part?

[patrickpoirier51]

@radfordi I think this script is asking a little too much for a single core. I experience too many timing issues including this:
TrackingManager: Stopping Manager - Cleaned 13145 non complete callbacks (onVideoFrame, onPoseFrame, onGyroFrame, onAccelerometerFrame, onControllerframe) - latency is too high!

I am presently building on a BananaPi Zero with quad-core Cortex A7 Allwinner H2+ processor running UBUNTU. http://wiki.banana-pi.org/Banana_Pi_BPI-M2_ZERO

Will update

[patrickpoirier51]

BananaPi Zero is working just fine

[MartyG-RealSense]

Though not related to the T265-specific theme of this discussion, I thought I would highlight a third party industrial version of the D435 from the company FRAMOS that I've come across that includes an M8 power connector.

#4769

[radfordi]

Very nice, @patrickpoirier51!

[lvale]

Though not related to the T265-specific theme of this discussion, I thought I would highlight a third party industrial version of the D435 from the company FRAMOS that I've come across that includes an M8 power connector.

#4769

Well, asking price ranges means it is out of the amateur/hobby user....

[MartyG-RealSense]

@lvale Yes, there is usually an old saying about luxury goods: "If you have to ask the price, you can't afford it". On the other hand, FRAMOS use a quote based system for every component on their store, even the ones we know the average retail price of such as the Intel models of RealSense camera, so it won't necessarily be a very large excess over and above the standard D435.

[patrickpoirier51]

@radfordi
Its flying :-)
A 330 mm size is much ''friendlier'' for garage experiments than my original Q450.
BananaPi makes a nice little companion computer, the complete sensor unit is around 150 Grams, including cables, Ubec & Mini USB HUb

https://www.youtube.com/watch?v=aQcRiFnSpGA

[MartyG-RealSense]

@patrickpoirier51 Congratulations, that's a beautiful vehicle! :)

[tropicdragon]

@patrickpoirier51

That looks really cool! I know you are using the BananaPi now, but it seems that you had it working with the Raspberry Pi Zero before, I am wondering if you ever had an issue with the RP Zero not detecting that the T265 is connected? I am currently trying to get something working, but can't seem to find a solution to this issue. Was wondering if you had any suggestions?

[patrickpoirier51]

@tropicdragon the detection issue has been documented, but I think they closed the thread without providing a solution for the moment.

Look at my hack to resolve the issue wit a hardware reset upon system loading:

#4681 (comment)

[tropicdragon]

@patrickpoirier51

Good to know! Thank you! #4681 seems to more or less describe the issues I am having.

Also, do you happen to have the shell script mentioned in the fourth bullet point? And would you be willing to share that?

[patrickpoirier51]

Well, the user space GPIO mapping and commands
https://raspberrypi-aa.github.io/session2/bash.html

[yputra]

@patrickpoirier51 Wow, this is such a good project you have done, thanks a lot for sharing all the details.
I am trying to do a similar one, but with Raspberry Pi 4, T265 and D435 as the mapping device, however it still has the issue mentioned on #5153 ... I wish it could be quite stable for a long run (20 minutes) constantly without this issue. Do you have any experience with something similar with #5153 ?

[patrickpoirier51]

@yputra I experienced this problem once during a test and it was related to a loose connection of the USB cable. I then used hot glue to secure the connections and it never reoccurred. Reading the issue, it seems some units are really marginal , hope they will release firmware that address this pattern

[Andrew5c]

@radfordi Thanks for your guidance.
We are running ArduPilot http://ardupilot.org/dev/index.html on the CUBE.

The USB port is could be used but I would require some work as there is not Host side implementation of the OTG at the moment. A method for supply power to T265 is necessary, so we could build a Power Injector cable assembly. Please note that all the above are not necessary with an auxiliary port on the T265.

Nonetheless, your idea of making a minimal example of using T265 on top of libusb (i.e. without all of librealsense) seems quite appealing as we could reduce the footprint (and associated weight) by using a much more compact Companion Computer like the RPI Zero.

Please let us know when your example is available so we can test and and ultimately provide the community with an image than can be flashed on a SD.

Here is an example of a RPI-Zero and a T265 , just need the USB-Injector to make a ready to fly VIO system ;-)

Hi, thanks for your project share. I am trying to do a similar project, which use the same T265 camera, but the processor I used is Respberry-3B, can I use the 3B to realize the UAV's spot hover? thanks!!

[patrickpoirier51]

Hello
Yes you can, but because of the USB2 you will only get the pose data and this ok for navigation

[Andrew5c]

Hello
Yes you can, but because of the USB2 you will only get the pose data and this ok for navigation

Hi，thanks for your reply timely. Now I want to use the NVIDIA's Jetson Nano as the processor, which has higher performance . Aim to realise the UAV's spot hover. but the question is, should I run SLAM on the Nano ? or just read the data from the T265? and T265 will run SLAM in itself? thanks!

[zainmehdi]

@patrickpoirier51 Thanks for all the useful information. I was wondering how did you change the frequency for pose values. Is there any function for that in librealsense ? I am planning to acquire pose at 10hz frequency ? Did you use a timer ?

[MartyG-RealSense]

@zainmehdi This is a potential workaround rather than an official feature, but in September 2019, one user's method for a custom frequency for the accelerometer that worked for them was to develop a RealSense ROS node in C++ that is controlled by a rate (they tried creating a node in Python but found it ineffective).

The link below provides an explanation about 'Rates' in ROS.

https://roboticsbackend.com/ros-rate-roscpy-roscpp/

[zainmehdi]

@MartyG-RealSense Thanks . I am developing in qt so I dont have ros rate option available. I will try with a timer and share my results.

[neilyoung]

@Andrew5c The T265 only provided pose, so in other words, it already does the localization. It will not do any mapping, so it will not do SLAM, but only Localization. However, the localization (running in the T265) is already enough for UAV hovering, if you are not building any mapping.

Either I misunderstand this or it is wrong. The T265 does mapping. And of course it does SLAM :)

[patrickpoirier51]

@zainmehdi we are extracting pose using a python script and set set desired frequency.
Tale a look at the very well documented experiments from @thien94 here
https://github.com/thien94/vision_to_mavros

[yputra]

@Andrew5c The T265 only provided pose, so in other words, it already does the localization. It will not do any mapping, so it will not do SLAM, but only Localization. However, the localization (running in the T265) is already enough for UAV hovering, if you are not building any mapping.

Either I misunderstand this or it is wrong. The T265 does mapping. And of course it does SLAM :)

I am sorry I worded wrongly, and I am not sure why I wrote it that way, anyway it does SLAM inside T265, and you can disable the mapping if you want to get only the pose for hovering or even simple movement in reference to current position of the drone

[neilyoung]

@yputra Interesting. I'm not aware of how you can disable mapping. Would you mind to share?

[neilyoung]

@patrickpoirier51 Thanks for all the useful information. I was wondering how did you change the frequency for pose values. Is there any function for that in librealsense ? I am planning to acquire pose at 10hz frequency ? Did you use a timer ?

Basically the solution is pretty simple: Just take every 20th pose frame and ignore the rest. Since with bad USB your client can miss some frames I would just set the number of the frame, which is the next to be processed, to the current number plus 20 and then take the next, which has a number greater or equal that threshold. Works pretty ok, not superior, but ok. I'm using every 5th pose to achieve 40 fps.

[patrickpoirier51]

@neilyoung as for SLAM options you can refer to this branch of @thien94 work:
https://github.com/thien94/vision_to_mavros/blob/SLAM_options/scripts/t265_to_mavlink.py#L177

[neilyoung]

@patrickpoirier51 Ah, very nice, Patrick. Thanks for the hint. I never "messed" with these options :)

[zainmehdi]

@patrickpoirier51 Thanks for all the useful information. I was wondering how did you change the frequency for pose values. Is there any function for that in librealsense ? I am planning to acquire pose at 10hz frequency ? Did you use a timer ?

Basically the solution is pretty simple: Just take every 20th pose frame and ignore the rest. Since with bad USB your client can miss some frames I would just set the number of the frame, which is the next to be processed, to the current number plus 20 and then take the next, which has a number greater or equal that threshold. Works pretty ok, not superior, but ok. I'm using every 5th pose to achieve 40 fps.

Thanks @neilyoung . I will try your approach as well however I was able to achieve my desired frequency using a qt timer that fires every 100 ms to capture pose and frames.

[bovlb]

I was thinking of making a minimal example of using T265 (pose only) on top of libusb (i.e. without all of librealsense) just to help bootstrap this kind of application.

@radfordi Did you ever get anywhere with this idea?

[radfordi]

@bovlb, yes, I did write a simple version. I'm working on getting it released. In the meantime, we also massively simplified (#5213) librealsense's interface to the T265. It's now more portable (it works on Mac and it should be very easy to port to Android), more robust, and the code is much easier to follow.

[bovlb]

@radfordi Thanks. I've been trying to get it working on the Roborio.

[zbynekwinkler]

To return to the original request - having UART port on something like T265 would be great for integration on drones and other robots. It would indeed very much simplify our life. Having a request-response protocol where one could ask for the current pose estimate (pose frame in librealsense speak) would be enough (equivalent of wait_for_frame and poll_for_frame to get either the next frame or the last frame would be even better - the first would wait for the new frame while the second would immediately respond with the last frame).

Think of connecting to Arduinos instead of full-fledged PCs. More in the sense of openmv that allows easy connection to a microcontroller.

[germal]

@patrickpoirier51 where is it possible to find the T265 bracket mount and the 90 degrees cable in the picture of your drone ?
Thank you !

[patrickpoirier51]

Hello @germal
The bracket is 3D printed GoPRO mount and the angled USB C cable you can find on AMAZON

[germal]

Hello @patrickpoirier51
Thank you, I found the cable , could you please send the stl file or a link to it ? I see that is really well suited for the t265 !

[patrickpoirier51]

Just type in google thingiverse and gopro ;-)

[germal]

@patrickpoirier51 I can find joints but not that particular bracket .

[patrickpoirier51]

LOL !!!!! Do you want me to print them for you :-)

https://www.thingiverse.com/thing:2255430
https://www.thingiverse.com/thing:3483497

[germal]

I am actually a disaster in mechanics : when I drew circumferences my teacher was used to told me that he didn't like eggs :-)
Thank you @patrickpoirier51 !

[jihadrahmawan]

BananaPi Zero is working just fine

hi @patrickpoirier51 , can i see your LibRealsense installation process? and what kind of your OS do you use on the BananaPI Zero? Thank you.