Support glTF meshes

[johnwason]

A new mesh format has been released and is rapidly gaining traction:

https://en.wikipedia.org/wiki/GlTF

https://blog.ybalrid.info/2018/07/why-gltf-2-0-is-awesome/

It appears to be supported by assimp. This new format is intended to fix the problems with collada, stl, etc.

[gavanderhoorn]

One more standard to rule them all :)

When I looked at it some years ago it was almost as if someone had dusted off VRML and gave it a new name.

[johnwason]

I guess the file types will continue to proliferate until one isn't terrible...

This one apparently matches the format used by WebGL, with the goal of minimizing the processing required to load into the GPU.

[Levi-Armstrong]

If it is supported by assimp then the file should get processed if added to the urdf.

[johnwason]

An interesting article discussing glTF from the Godot team:

https://godotengine.org/article/we-should-all-use-gltf-20-export-3d-assets-game-engines

glTF 2.0 supports vendor extensions:

https://github.com/KhronosGroup/glTF/tree/master/extensions

Would it make sense to add robot specific extensions to glTF as a replacement for SDF and URDF?

[gavanderhoorn]

It would be interesting, but this starts to sound rather like Collada. That also supports extension points.

A good example of a tool which uses extensions to Collada heavily for robotics purposes is OpenRAVE.

Problem with "vendor extensions" is that most of the time, vendors don't support extensions of other vendors (for whichever reasons). You can then get into the situation where loading a mesh with extension vendor_A and serialising it again with a tool from vendor_B would result in loss of the elements in the mesh which use/depend on the extension. As vendor_B has decided not to add support for that extension.

I've not checked whether Khronos has made provisions for this in the standard (ie: require in==out for unsupported extensions), so it could be they've plugged this hole, but at least for Collada, what I describe above happened.

---

And it was exactly the extension system which was one of the biggest issues with Collada when it was discussed as a potential successor to urdf.

[gavanderhoorn]

We could of course decide we care about none of that and still use glTF. I did not want to imply that should not be done.

But if the goal is to find a format which can be used both as urdf / sdf and what have we can (ie: by humans, in OSS projects, etc) and can be reliably imported and exported from commercial tools, we have to be careful.

[johnwason]

@gavanderhoorn Khronos is collecting extensions, reviewing, and storing them in the main repo:

https://github.com/KhronosGroup/glTF/tree/master/extensions

There is a dependencies statement in the file to indicate which extensions are required.

It also appears Solidworks has an exporter for glTF now: https://blogs.solidworks.com/solidworksblog/2019/01/introducing-solidworks-extended-reality.html

Blender and Autodesk also support it.

I am not sure about how human readable the file will be. This is something that will need to be investigated before using it.

[gavanderhoorn]

Seems this discussion is something which could be interesting in the context of ros/urdf#34 as well.

[gavanderhoorn]

And I just noticed Blender 2.8 and newer support gTLF.

[johnwason]

There are a few existing proposals to create a glTF based replacement for URDF and SDF:

• https://ieeexplore.ieee.org/abstract/document/8971739
• https://gist.github.com/mikedh/ce09d1bf83767fefded25be519a61564

I am probably going to try to put together a working group to see what interest there is. I don't think either URDF or SDF are going to work for my current projects.

[gavanderhoorn]

I don't think either URDF or SDF are going to work for my current projects.

I'd suggest to try and reuse something existing if at all possible, for obvious reasons.

[johnwason]

We are planning on using glTF with extensions

[gavanderhoorn]

There are a few existing proposals to create a glTF based replacement for URDF and SDF:

• https://ieeexplore.ieee.org/abstract/document/8971739
• https://gist.github.com/mikedh/ce09d1bf83767fefded25be519a61564

If we're honest, I'd hardly call these proposals. The first is a paper by two researchers and the second is a gist with some good ideas, but there doesn't appear to be any implementation available (and the gist is already almost 1.5 years old).

You seem to have started KhronosGroup/glTF#1816. Is that the "glTF with extensions" you mention?

[johnwason]

Yeah, I asked @wlawler45 to take a look. The amount required to add robot information to glTF is fairly minimal. We are going to come up with a formal proposal over the next few weeks/months. The infrastructure to support glTF is already widely available (far more than even COLLADA), so it should not be all that difficult.

[gavanderhoorn]

There are now a few different discussions going on in parallel.

I posted a link to this issue here on ROS Discourse: Web-based URDF visualization tool and library opensourced from NASA JPL.

@gkjohnson responded there. Perhaps he'd like to post a few comments here as well.

[Levi-Armstrong]

This is a bit of what I'm afraid of. I think a distinction should be made between reusable robot model and the context / scene in which it's operating. My feeling is that URDF should represent a built robot and the properties associated with it -- sensors, joints, geometry, etc -- and should be kept separate from the environment in which said robot might be operating or simulated because it can be operated elsewhere. A scene should be defined and it should point to the assets representing the robots operating in it. To that end it feels that things like an occupancy map, point clouds, or terrain should be represented in the "scene" description rather than the robot description. Perhaps SDF is that scene format (I'm trying to find some time to learn abit more about it) but there seem to be dissenting opinions, as well. Maybe creating a more distilled or general version of SDF (or moving the existing format in that direction) would be good.

I agree and what you are describing is more or less the SDF format. Where in this case the model element in SDF is the the URDF as you described it above. Then the world and light tag is the scene representation.

[johnwason]

I created a repo for a potential working group: https://github.com/wasontech/robot-description-wg

For now I am using my company's organization. This may move to RPI or ARM depending on how a formal group is organized. Please create one issue for each feature request so we can discuss them individually.

[johnwason]

Please watch https://github.com/wasontech/robot-description-wg if you wish to be notified of new proposed changes.

[clalancette]

Please watch https://github.com/wasontech/robot-description-wg if you wish to be notified of new proposed changes.

I'm also going to setup an in-person call and announce a working group on discourse. I just have been running out of time every day. I'll post here once I've set something up.

[johnwason]

@clalancette I think it may be better to create an initial proposal first with a small group before announcing to the broader community. These things have a tendency to get bogged down with too many cooks in the kitchen.

[johnwason]

Is there a charter template for ROS 2 working groups?

[johnwason]

Initial charter added to readme: https://github.com/wasontech/robot-description-wg/blob/master/README.md

A two step process is suggested to help deal with deadlines for our active projects.

[clalancette]

@clalancette I think it may be better to create an initial proposal first with a small group before announcing to the broader community. These things have a tendency to get bogged down with too many cooks in the kitchen.

While I somewhat agree, there is a lot of duplication going on here. Which means there are already a lot of cooks, they are just in different kitchens and not talking to each other. That's why I'd rather have all of the people who care about this get together at least once. If the outcome of that is a small group that goes away and comes back with something, that's fine.

[johnwason]

@clalancette that seems reasonable. If we want to do a community-wide working group, it should probably be done through ROS rather than me running it (I am a one-man shop). That unfortunately does not solve the active projects' near-term needs.

[johnwason]

The Open Source Teach Pendant project that was the motivation for this project has finished its initial funded development phase. It was discussed in #629 .

The repositories for the Tesseract extensions:

https://github.com/pyri-project/pyri-tesseract-planning-browser https://github.com/pyri-project/pyri-tesseract-planning

This is currently using the Babylon.js version of the viewer, and the modified URDF file format loader that Tesseract uses. I'd like to be able to have the different devices provide the URDF and resource files automatically. Right now this isn't really possible using URDF.

As part of the user interface, I used the tesseract_python_viewer as a simple proof-of-concept Tesseract extension. I haven't had the time to develop the Babylon.js viewer further, and will probably wind up switching to another viewer which is more developed. @gkjohnson Are you still actively working on your Three.js viewer? Have you tried out the improved webxr support recently?

[gkjohnson]

@gkjohnson Are you still actively working on your Three.js viewer? Have you tried out the improved webxr support recently?

The open sourced URDFLoader project is intended more to enable simple loading and manipulation of URDF files (a la other three.js file format loaders like GLTFLoader). The example viewers in that repo are really just intended to be a simple of demonstration of how to use the loader. A more complex viewer with multiple robots. interactions, or visualizations would be up to the dependent application to implement. We have been building more robust web-based visualizations for ROS and other robotics platforms using URDFLoader internally but unfortunately that's not open source at the moment.

Regarding WebXR awhile ago I updated the VR demo in that project to include the ability to manipulate loaded robot with controllers.

I'd like to be able to have the different devices provide the URDF and resource files automatically. Right now this isn't really possible using URDF.

I don't quite follow this. The infrastructure for loading and transferring resources to the client should be independent from the file format. Maybe I'm not understanding the limitation?

[sancelot]

In my opinion, this is the way to go. GLTF is today mandatory

[sancelot]

GLTF is now becoming widely sprayed . I already made some industrial 3D realtime applications .
To achieve it , web technology is really a must at the moment (babylonjs/threejs/ enable3d.......) and provide many advantages to code it once, and run it anywhere.....

some C/C++ api exists , but they either only permit to display a gltf file or only manipulate gltf data, but not both.

[johnwason]

The main problem with URDF right now is it doesn't support composition of different models like SDF does. For instance, there is no concept of a "model" and instances of the model, and there is no way to easily add or remove robots. For instance, creating an editor that you can insert, drag around, and connect robots/accessories isn't really possible. There are also some missing features, and no easy way to add extensibility.

Xacro is the current solution, but all context is lost after it is run. Also, it is inherently unsecurable and requires Python. (Unless there is a C++ implementation I haven't found)

[gkjohnson]

Xacro is the current solution, but all context is lost after it is run. Also, it is inherently unsecurable and requires Python. (Unless there is a C++ implementation I haven't found

I have published this Javascript implementation of a xacro processor which uses the expr-eval package to run the expressions so it doesn't run completely arbitrary JS. Of course as far as I understand the original xacro package can evaluate any python expression so this won't work on some of the more complex xacro files but you should be able to get reasonable expressions to work.

[johnwason]

We'd need a C++ version as well eventually. Have you tried out brython? It looks like an interesting way to share codebases between Python and Javascript without the overhead of pyodide.

[gkjohnson]

I've seen Brython but I think in most cases complex python expressions shouldn't be needed for describing a kinematic system so I chose a simpler evaluation system which makes it more portable and keeps the size down, as well. If you're in an environment where you can and want run python expressions arbitrarily (brython or other in a browser, desktop environments with python child process in node / electron) you can override the expression evaluation logic through an option to call out to whatever service or library you want. The same goes for any of the other rospack commands that are used within xacro files.

I haven't seen a C++ version of xacro anywhere, though.

[gavanderhoorn]

I haven't seen a C++ version of xacro anywhere, though.

with all the support for dynamic typing + Python expression evaluation, a C++ implementation of xacro would seem like it would be quite complex to get to work.

[sancelot]

From my viewpoint it is a complete nonsense needing to use xacro to generate urdf files for industrial robots (eg fanuc). But I can understand it is needed for other purposes.

Regarding xacro macro and many of other python utilities . There is no need to complicate more the things. Packaging xacro as binary would be enough. And regarding windows platform, this is why I started this project https://github.com/sancelot/ROS2_MSYS2_MINGW64
aiming to be able to package most of the utilities as binary windows executable, using tools like pyinstaller.
The same packaging system will work for linux

[johnwason]

I have been seeing more projects using jinja2 for templating, and there are many implementations available for various languages. I mention it in #622. It is a lot more powerful than xacro, but would need a few extra functions beyond what it has built in to be used like xacro. Jinja2 wouldn't solve all of the problems with the current URDF, but it would help.

[johnwason]

I moved the tesseract_viewer_python module over to use threejs instead of babylonjs. I haven't had time to work on the babylonjs viewer, and the threejs library has improved. The main missing feature in threejs is a good VR teleport control. More discussion in the pull request: tesseract-robotics/tesseract_python#21