Implement Skeleton re-targeting node

[reduz]

Describe the project you are working on

Godot

Describe the problem or limitation you are having in your project

The only way to re-use animations is use compatible skeletons with similarly named bones and proportions. This can be significantly limiting for sharing animations or using animations from common sources such as Mixamo.

While in-engine re-targeting is generally not a technique advised for production (as it makes the amount of bones you can use to animate rather limiting, thus makes rigging looks worse, for reference most studios do this within the 3D DCC together with IK), there are still cases where it makes a lot of sense, such as body tracking, prototyping, or simply Indie development where creating custom animations for models can be expensive.

Note, this is an actual implementation proposal based on #2619, so it supersedes it.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

The proposal here is to implement re-targeting as a special node. Mostly to make Skeleton less bloated and to reduce the need to make children scenes editable in order to configure them.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

A new SkeletonRetarget node will be provided by the engine. This node will have the following properties:

• source : NodePath
• target : NodePath
• retargeter : Retargeter (resource)

Retargeter is a resource that does the retargeting logic. It is a virtual abstract class that does no logic, but it can be inherited to implement a retargeter. Godot will provide a default implementation: HumanoidRetargeter which will come with a default humanoid skeleton.

Editing HumanoidRetargeter will be similar to Unity/Maya/etc., except that you must set the source bone and destination bone.

Given the Retargeter is a resource, it will be possible to save it to disk to use it in other scenes.

Eventually, it should be possible to provide other more precise retargeters with more complex models via GDExtension if desired.

If this enhancement will not be used often, can it be worked around with a few lines of script?

Animation is core

Is there a reason why this should be core and not an add-on in the asset library?

Animation is core

[fire]

Will post some extended uiux mockups for matching body armatures:

1. measuring tape measurements
2. put dots on a doll
3. automatic matching

[drcucaracha]

Hello, can I contribute an idea? What I think could be interesting and with very little complications would be to make a panel with a series of grooves in this case (BoneAttachment) and for the user to add the bones according to the name of each bone in the panel and it would be necessary to have a way of importing animated or non-animated skeletons that can be scaled and adapted to the model and for each complex part such as head, hands etc. make a sub-panel in case you want to add more bones or not that would cover the part of the non-standard bone names

image of doubtful origin but as an example I think the idea is understood

[fire]

Yes. I personally done 10-20 different character with the full body grooves in the Blender VRM addon. https://github.com/saturday06/VRM_IMPORTER_for_Blender

I am not happy with moving bones onto grooves.

Only the most stubborn rigger will go to the effort of making a common skeleton chart and doing a look up table to memorize the exact mapping.

[drcucaracha]

Hi, I'm not very excited about moving bones either but I can't think of another idea since the bones can have any name or they can use models already with bones from some downloaded site and the nomenclature will never be standard, the other could be a custom plugin in blender that renames the bones to use a predefined nomenclature in godot to load them automatically a kind of baking before exporting in pose T and have a skeleton already prepared with the same nomenclature which is not how the addon would know that bone is which to rename

[TokageItLab]

My thinking, RetargetNode will subscribe to Skeleton3D's PoseUpdate signal and override the pose when the signal fired.
However, I think we need to think about the order of processing when considering compatibility with IK and some addons. In some cases, it may be necessary to overwrite the pose as an internal process of Skeleton3D, just as IK does.

[GeorgeS2019]

It is great to see many improvement in 3D skeletal animation support in Godot recently.
Is there a plan for :

e.g. A complementary proposal/issue that completes the workflow from BVH motion file to Skeleton retargeting?

Is there a plan to make Godot part of the standard list of software for Motion Capture e.g. Rokoko Suit
e.g. Someone has made available Frabrik-IK to Godot

[TokageItLab]

I did a little experiment.

https://twitter.com/viridflow/status/1467679166137266177

As a result, I have two suggestions.

---

One is to apply RetargetNode as a child of Skeleton, like SkeletonIK. This is to make the RetargetNode commonly applicable not only for FK animations, but also for IK, etc. This and using intermediate rigs will help us implement some features such as motion tracking, as we can implement no dependency on Rest for IK.
However, it may need to be called from within Skeleton's set_bone_pose(), since it needs to be done after the application process of IK, etc. Additional verification is needed to see if the order in which it is called by signal is Tree dependent, etc. Possibly this problem may be unfounded.

Edited:
As for the above, it turns out that RetargetNode doesn't need to be a child of Skeleton, but we need to refactor the bone_pose_override (godotengine/godot#55840).

---

The second is to implement an intermediate track as a variation or option of the Pos/Rot/Scl track. In the current implementation, an extra Skeleton is always needed in the scene to use the RetargetNode. Actually, by pre-multiplying the source Skeleton with the animation values, the Skeleton is no longer needed in the Scene. This simplifies the management of animation resources when performing FK retargeting.
However, we will still need a map of bone relationships, so I will need to implement a GUI that allows us to easily create intermediate skeleton names and intermediate bone names for intermediate tracks, but I think it's worth doing.

[Zireael07]

Isn't this implemented in Godot 4.x now?

[TokageItLab]

Retarget in 4.0 still exists only in importer and retarget node is not implemented in core, but it now can be able to be implemented as a custom module.

[GeorgeS2019]

Retarget node as a custom module

@TokageItLab => any update for 2023?

[TokageItLab]

Now that https://godotengine.org/article/design-of-the-skeleton-modifier-3d/ has made it possible to implement Constraints, etc., we think it is worthwhile to start again with this as a realtime retarget.

Since having Retarget mode in Animation like godotengine/godot#56902 has already been rejected twice by @reduz before, there is no other way for a realtime retarget except to design it correctly with SkeletonModifier3D.

If it works, we can import Blender's Constraint settings.

---

Implement AnimationProvider class by extending SkeletonModifier

AnimationProvider has a Skeleton without Skin whose children are Override Rest.

It has the following enum and properties to set RetargetMode:

enum RetargetMode {
	RETARGET_MODE_GLOBAL, // default
	RETARGET_MODE_LOCAL,
	RETARGET_MODE_ABSOLUTE,
};

RetargetMode spines;
RetargetMode hands;
RetargetMode arms;
RetargetMode legs;

Please refer to the following link for an explanation of RetargetMode.

https://github.com/TokageItLab/realtime_retarget

Basically, RETARGET_MODE_GLOBAL is preferred, but RETARGET_MODE_ABSOLUTE looks better for fingers in some cases, and RETARGET_MODE_LOCAL is preferred for arms and legs in some cases.

This approach means we cannot blend the different RetargetModes, but there is nothing we can do about that since we cannot have a RetargetMode in the Animation. This is probably the maximum compromise.

---

Now importer has only the Override Rest option as a rest fixer, but we will use this as an enum, Don't Override Rest, Override Rest and Use Animation Provider.

If the Use Animation Provider option is used, skeletons are imported in the following tree:

%GeneralSkeleton (Skeleton3D, keep original rest)
└MeshInstance (MeshInstance)
└AnimationProvider (SkeletonModifier)
　└Skeleton (Skeleton3D, rest is overridden)

The most debatable thing here is that the Animation track path is %GeneralSkeleton/AnimationProvider/Skeleton instead of %GeneralSkeleton.

This means that animations imported with Use Animation Provider and animations imported with Override Rest will have different paths and we will not allow them to be mixed, but this may be a documentation issue, although we may be able to issue some kind of warning.

Other debatable points:

• Performance concerns about having two skeletons
  • As this is a skinless skeleton with only a Humanoid Bone, the performance penalty is not expected to be greater than a certain amount.
• Whether to allow external Skeletons to be specified
  • This is probably possible, but in practice it makes no sense. As explained in https://godotengine.org/article/design-of-the-skeleton-modifier-3d/, there are several stages in the Skeleton update process, and AnimationProvider only considers the result of set_bone_pose() (before deferred process). For example, if you want to apply an external Skeleton IK and retarget it, AnimationProvider will not get the pose if the IK extends a SkeletonModifier. Also, the root may be a %GeneralSkeleton due to the Import as Skeleton option, so when you import it, it must be in the hierarchy described above. We do not want any more confusion in the animation path due to external skeletons.
• This can cause problems for modules that rely on the HumanoidProfile for axes, as models that do not use Override Rest are imported in some cases.
  • I won't know without trying, but if AnimationProveider can get the pose of the SkeletonModifier of the Skeleton under AnimationProveider after processing, that problem could be solved. In that case it might make sense to allow an external skeleton.
  • It means that the GeneralSkeleton Deferred process must be performed after the Skeleton Deferred process.

Or, except that it is not intuitive, but reversing the hierarchy explained above may solve most of the problems except double Skeletons...

%GeneralSkeleton (Skeleton3D, rest is overridden)
└AnimationProvider (SkeletonModifier)
└Skeleton (Skeleton3D, keep original rest)
　└MeshInstance (MeshInstance)

In this case, the AnimationProvider will pass the pose of the parent Skeleton to the target Skeleton; The Skeleton Deferred process must be performed after the GeneralSkeleton Deferred process. It might be better to name the class PoseTransporter.

---

Thoughts? @fire @lyuma @reduz