Feature/visualization++ #802

S-Dafarra · 2021-01-22T11:36:28Z

It does the following:

It adds a camera animator to move the camera via the mouse
It fixes the aspect ratio to avoid strange effects when resizing
It adds an interface to add custom frames to the visualization
It fixes the floor grid colors
It adds an interface to have additional textures (helpful if you need to render to a different size with a different environment and to get the pixels of the corresponding image)
It fixes the visualizations of STL meshes.

…ation interface.

…t ratio. This bug was also affecting the camera animator, with the controls along x being mirrored.

S-Dafarra · 2021-01-22T11:48:03Z

src/visualization/src/Texture.cpp

+        for (size_t width = 0; width < textureDim.Width; ++width)
+        {
+            for (size_t height = 0; height < textureDim.Height; ++height)
+            {
+                pixelIrrlicht = irr::video::SColor(*(unsigned int*)(buffer + (height * pitch) + (width * bytes)));
+                pixels[i].width = width;
+                pixels[i].height = height;
+                pixels[i].r = pixelIrrlicht.getRed();
+                pixels[i].g = pixelIrrlicht.getGreen();
+                pixels[i].b = pixelIrrlicht.getBlue();
+                pixels[i].a = pixelIrrlicht.getAlpha();
+                ++i;
+            }
+        }


FYI, this is pretty inefficient. I would have liked to output directly buffer, but it should have been necessary to implement also the lock/unlock mechanism required by irrlicht. In addition, it was not clear how to pass the information on how to interpret that buffer. For RGB images, having the raw buffer may be handy with YARP images, thus avoiding two double for loops. In fact, irrlicht and YARP seem to store RGB images in the same way. It is not the same for RGBA. In fact, irrlicht uses ARGB instead.
Bottom line is, I did not find a better way to output the raw image. In any case, this has an effect only when the corresponding texture is large, or when getting the pixels of many textures.

traversaro · 2021-01-22T12:08:58Z

CMakeLists.txt

@@ -8,7 +8,7 @@

 cmake_minimum_required(VERSION 3.16)

-project(iDynTree VERSION 2.0.2
+project(iDynTree VERSION 2.0.3


Feel free to bump more (like 2.0.100), as 2.0.3 in the future could be be a patch release on the master branch.

Done in b10c4b8

traversaro · 2021-01-22T12:24:32Z

src/visualization/include/iDynTree/Visualizer.h

@@ -466,6 +576,41 @@ class IModelVisualization
    virtual Transform getWorldLinkTransform(const LinkIndex& linkIndex) = 0;
 };

+class ITexture


Can you add here or in some other related interface what you mean by "Texture"? In 3D Graphics texture is typically used to refer to 2D images that is then mapped to the surface of a 3D model to describe its appearance (see https://en.wikipedia.org/wiki/Texture_mapping). In this context instead if I understood correctly the texture is a sort of "render target" (see the second meaning in https://www.khronos.org/opengl/wiki/Texture). As this could be confusing for users, could you clarify this? Thanks!

Yes, the correct interpretation would be that the texture is a different target for the renderer. Do you think that adding some documentation may be sufficient, or you about a better naming?

As you prefer, I think that even just documentation is ok.

I have updated the documentation in b7267fc

traversaro · 2021-01-22T12:28:28Z

src/visualization/src/IrrlichtUtils.h

@@ -73,7 +66,25 @@ inline iDynTree::Rotation irr2idyntree_rot(const irr::core::vector3df & rotIrr)

 inline const irr::core::vector3df idyntree2irr_rot(const iDynTree::Rotation & rot)
 {
-    return idyntree2irr_rpy(rot.asRPY());
+    irr::core::matrix4 irrTransform(irr::core::matrix4::EM4CONST_IDENTITY);
+    //Transposing the matrix since irrlicht uses left handed frames


Can you elaborate a bit more on this? I thought that the rendering problem of STL meshes was solved by changing the scale of the X variable, instead also this is necessary? Did you tested this with the iCub 2.5 DAE models after this changes?

I reverted the changes in 0a6f821.
Regarding the visualization, I run idyntree-model-view with:

the `iCubGenova09`` urdf

the iCubGenova04 urdf

a custom iCubGenova04 urdf with dae meshes

Ok thanks! However, if I recall correctly is that the problem with the stl meshes on iCub3 (the one that we imagine was fixed by the STL load fix) only appeared when the legs or arm were moved from the zero position, so after you reverted the change to the rotation did you check if the visualization is still working as expected?

Here the result when reading the joints from the control boards:

iCubGenova09 urdf

iCubGenova04 urdf

iCubGenova04 urdf with custom DAE meshes

traversaro · 2021-01-22T12:30:42Z

src/visualization/include/iDynTree/Visualizer.h

+     * Remember to call draw() first.
+     * @param pixels The output pixels
+     * @return True in case of success, false otherwise
+     */


Can you clarify here:

the size of std::vector

if the pixel are stored in row major or col major order?

Now that this docs seems to be duplicated also in the concrete Texture implementation, so either update also that one, or remove it.

I have updated the docs in b7267fc and removed the duplicated documentation in d337204.

traversaro

Some comments, but overall it seems great, thanks!

traversaro · 2021-01-22T12:49:37Z

By the way, this new features (especially the mouse control and the frames) could be used to enhance idyntree-model-view ( https://github.com/robotology/idyntree/blob/master/src/tools/idyntree-model-view.cpp ) and make it a quick and dirty tool for easy inspection of URDF models frame placement. fyi @Nicogene

S-Dafarra · 2021-01-22T18:49:42Z

By the way, this new features (especially the mouse control and the frames) could be used to enhance idyntree-model-view ( https://github.com/robotology/idyntree/blob/master/src/tools/idyntree-model-view.cpp ) and make it a quick and dirty tool for easy inspection of URDF models frame placement. fyi @Nicogene

I enabled the mouse control for idyntree-model-view in 2e71c6f.

prashanthr05 · 2021-01-25T08:41:16Z

src/visualization/src/FrameVisualization.h

+         * @brief Add a frame in the visualization
+         * @return The frame index.
+         */
+        virtual size_t addFrame(const Transform& transformation, double arrowLength = 1.0) final;


Maybe it would also be convenient to have a method that removes a frame?

I have tried adding the method to remove a frame, but the logic behind was not the best. Since I am returning an index as an identifier, I do not like what happens when you remove a frame. Should that index be reused or not? Also, with the getNumberOfFrames method you can understand whether an index is valid or not. But if you remove one frame, this does not hold anymore as the total number of frames will be lower or equal than the maximum index. Hence, for the time being, I preferred to allow controlling the visibility of a frame with 98e804a. The net result is the same and the logic behind remains straightforward.

traversaro · 2021-01-26T09:59:42Z

Ok for me, waiting for @prashanthr05 approval as well.

@S-Dafarra it is ok for me to Squash and Merge or you want to revise the commits?

S-Dafarra · 2021-01-26T10:08:16Z

Ok for me, waiting for @prashanthr05 approval as well.

@S-Dafarra it is ok for me to Squash and Merge or you want to revise the commits?

It is ok to squash for me!

prashanthr05

In general, ok for me. I just had one clarification request in the CameraAnimator class.

prashanthr05 · 2021-01-26T11:20:17Z

src/visualization/src/CameraAnimator.cpp

+    initialTransformation(0,0) = xAxis.X;
+    initialTransformation(0,1) = xAxis.Y;
+    initialTransformation(0,2) = xAxis.Z;
+
+    initialTransformation(1,0) = yAxis.X;
+    initialTransformation(1,1) = yAxis.Y;
+    initialTransformation(1,2) = yAxis.Z;
+
+    initialTransformation(2,0) = zAxis.X;
+    initialTransformation(2,1) = zAxis.Y;
+    initialTransformation(2,2) = zAxis.Z;


Could you just document this row-major order stacking for the axis vectors and why we use this ordering and the difference from column-major ordering? (rotation and its inverse)

I have added the relevant comment in e20efda

idyntree/src/visualization/src/CameraAnimator.cpp

Lines 172 to 180 in e20efda

// Getting the rotation directly from camera results in undesired rotations when getting close to the singularity

// due to the fact that irrlicht stores rotations using RPY. As a consequence, we reconstruct the camera frame by

// having the z-axis parallel to the line connecting the camera position to the target position. The x-axis is

// reconstructed from the view frustrum. The view frustrum is the region of space that appears in the camera. It

// is basically a truncated rectangular pyramid. It is delimited by 8 points. We use the bottom left and top

// left points in the "far" plane to define the up direction, the x-axis. The y-axis is obtained by cross-product

// between the other two axes, obtaining a right-handed frame. Since irrlicht uses left-handed frames, the obtained

// frame is converted into a left-handed frame by taking its transpose (a left hand rotation corresponds to the

// inverse of a right hand rotation since the axis is the same, but the angle is on the opposite direction).

Thank you. Very clear!

Indeed, that transpose caused me a lot of thinking in the first place. Initially, it was not transposed, but the visualization was clearly wrong. So I thought I was doing some mistakes in the interpretation, I tried the transpose and everything went fine. Later on, when I was debugging a problem with the stl meshes being in the wrong position, I discovered this thing about irrlicht using left-handed frames, and then everything made sense 😅

S-Dafarra added 20 commits January 12, 2021 18:43

Added visualization to texture.

461ecfe

Allow changing floor grid color.

272c5d3

Added first draft of camera animator and fixed aspect ratio.

0850b12

Added basic zooming feature.

5a8c891

Added some pan control.

3619508

Fixed camera animator.

9be123b

Added possibility to turn on/off camera control. Added frame visualiz…

897663f

…ation interface.

Fixed DummyCamera.

21531aa

Added material flag to fix stl visualization.

b4babbb

Added textureEnvironment and possibility to set grid color.

7d2f42e

Fixed color visualization of the floor grid.

e70a8c0

Added test for the visualizer additional texture.

bb096f6

Added frames visualization.

543c37e

Avoid the camera to use left-hand convention when resetting the aspec…

f74eab6

…t ratio. This bug was also affecting the camera animator, with the controls along x being mirrored.

Improved frame visualization test.

913c415

Moved CameraAnimator interface to Visualizer header.

129b315

Added virtual destructor to IFrameVisualization.

d19b06d

Added textures interface.

5c9e2a3

Cleaned Visualizer interface removing old texture related methods.

5a0e3cd

Fixed stl visualization by mirroring it on the x axis.

2e4b8d5

S-Dafarra requested review from traversaro, prashanthr05 and GiulioRomualdi January 22, 2021 11:36

S-Dafarra self-assigned this Jan 22, 2021

Updated Changelog.

478b3de

S-Dafarra commented Jan 22, 2021

View reviewed changes

traversaro reviewed Jan 22, 2021

View reviewed changes

traversaro requested changes Jan 22, 2021

View reviewed changes

S-Dafarra added 5 commits January 22, 2021 19:11

Changed the version number after the visualizer++ PR.

b10c4b8

Improved the documentation of iTexture.

b7267fc

Restore conversions of rotations.

0a6f821

Enable mouse control on idyntree-model-view.

2e71c6f

Removed duplicated documentation.

d337204

prashanthr05 reviewed Jan 25, 2021

View reviewed changes

Allow changing the visibility of a frame.

98e804a

traversaro approved these changes Jan 26, 2021

View reviewed changes

prashanthr05 approved these changes Jan 26, 2021

View reviewed changes

Added comment on how the camera orientation is computed.

e20efda

traversaro merged commit 4bc88cf into robotology:devel Jan 26, 2021

lrapetti mentioned this pull request Feb 1, 2021

Add mouse camera control robotology/human-dynamics-estimation#233

Merged

Giulero mentioned this pull request Feb 25, 2021

Fix obj visualization #833

Merged

traversaro mentioned this pull request Feb 21, 2024

Add visualization of a subset of frames in idyntree-model-view #1155

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/visualization++ #802

Feature/visualization++ #802

S-Dafarra commented Jan 22, 2021

S-Dafarra Jan 22, 2021

traversaro Jan 22, 2021

S-Dafarra Jan 22, 2021

traversaro Jan 22, 2021

S-Dafarra Jan 22, 2021

traversaro Jan 22, 2021

S-Dafarra Jan 22, 2021

traversaro Jan 22, 2021

S-Dafarra Jan 22, 2021 •

edited

Loading

traversaro Jan 25, 2021

S-Dafarra Jan 26, 2021

traversaro Jan 26, 2021

traversaro Jan 22, 2021

S-Dafarra Jan 22, 2021

traversaro left a comment

traversaro commented Jan 22, 2021

S-Dafarra commented Jan 22, 2021

prashanthr05 Jan 25, 2021

S-Dafarra Jan 26, 2021

prashanthr05 Jan 26, 2021

traversaro commented Jan 26, 2021

S-Dafarra commented Jan 26, 2021

prashanthr05 left a comment

prashanthr05 Jan 26, 2021

S-Dafarra Jan 26, 2021

prashanthr05 Jan 26, 2021

S-Dafarra Jan 26, 2021

	// Getting the rotation directly from camera results in undesired rotations when getting close to the singularity
	// due to the fact that irrlicht stores rotations using RPY. As a consequence, we reconstruct the camera frame by
	// having the z-axis parallel to the line connecting the camera position to the target position. The x-axis is
	// reconstructed from the view frustrum. The view frustrum is the region of space that appears in the camera. It
	// is basically a truncated rectangular pyramid. It is delimited by 8 points. We use the bottom left and top
	// left points in the "far" plane to define the up direction, the x-axis. The y-axis is obtained by cross-product
	// between the other two axes, obtaining a right-handed frame. Since irrlicht uses left-handed frames, the obtained
	// frame is converted into a left-handed frame by taking its transpose (a left hand rotation corresponds to the
	// inverse of a right hand rotation since the axis is the same, but the angle is on the opposite direction).

Feature/visualization++ #802

Feature/visualization++ #802

Conversation

S-Dafarra commented Jan 22, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

S-Dafarra Jan 22, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

traversaro left a comment

Choose a reason for hiding this comment

traversaro commented Jan 22, 2021

S-Dafarra commented Jan 22, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

traversaro commented Jan 26, 2021

S-Dafarra commented Jan 26, 2021

prashanthr05 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

S-Dafarra Jan 22, 2021 •

edited

Loading