Metahuman - Retargeting

[MrTeaRexx]

Greetings ^_^, I'm just wanting to know is your version compatible with metahuman? reason im asking is due to your skeleton being different from the blueprint and dyanikoglu/ALS-Community versions as they use the same skeleton setup. Ive been trying to get it to work with yours but every time i add the appropriate VB's and sockets and do the re-targeting it crashes the engine.

[Sixze]

Crashes after retargeting are a common problem. First try to retarget without virtual bones, then restart Unreal, and only then add them.

[MrTeaRexx]

sure once i test it and give you a copy

[MrTeaRexx]

Currently in the testing phase

[MrTeaRexx]

unfortunately this is a no go due to metahumans modularity, unless they add the Ik's themselves as a option, its not viable since every model part, shirts, pants, shoes, body etc would need these changes and then would need to be done for the different size types short, average tall and the male versions. then if they bring new models out you would have to do those changes to them as well. epic doesn't like it when you try to add the modified skeletons. but i found a way around it but it has major issues. Just going to delete the VB bones and sockets and just re-target without it. the best option for metahuman to be compatible with this plugin is to have a separate control rig and maybe blueprint version skeleton ALS that uses VB for the ik system as ill be doing this as well for my project. I tried my best but not much i could do unfortunately

[Sixze]

Have you tried doing this:

1. Take all Metahuman skeletons and add IK bones to them.
2. Replace all standard skeletons with modified ones (as written here: https://docs.unrealengine.com/4.26/en-US/Basics/ContentBrowser/AssetConsolidationTool/)
3. Refresh the rig hierarchy of any existing Control Rig using the new skeletons.
4. Remove all bones and sockets from the ALS skeleton.
5. Retarget ALS using the metahuman_base_skel skeleton.
6. Add virtual bones and sockets to metahuman_base_skel.
7. Modify CR_Als to use the metahuman_base_skel skeleton.

In theory, that's all. Different proportions of characters should be automatically handled by runtime retargeting, at least as long as they have a similar skeletons.

[MrTeaRexx]

tried that method brakes everything in the metahuman structures and also every time you re-import new metas the references get reset.
I made a video for you explaining, how i did it but their are some issues of the IK not lining up and the control rig fusing bones and other stuff which is mentioned in the video. Im still learning how to do the control rig in ue4 so my knowledge in that area is limited as epic want you to make controls for the skeleton similar to maya and then use those controls to drive the IK https://www.youtube.com/watch?v=y2WzNvJZk0E which is the way im setting up mine so that way if new animations are required i can make them without having to switch back to maya.

Video: https://www.youtube.com/watch?v=Ii5UpN6DYlc
Source files: https://1drv.ms/u/s!AgYK1WtqwMW1kgxMoiqzWzX_YpOc?e=ozan1I

[Sixze]

Okay, on the weekend I will try to retarget MetaНuman myself and write more detailed instructions if it works out, but can you please take all these skeletons from the MetaНumans sample project, add IK bones, and give them to me?

[MrTeaRexx]

Anything face related you wont need as the body has bone neck 01, the rig in the face is a nightmare cause as its rigged and anim bp'ed for livelink which controls it and fuses with the body and i don't want to touch it, the skye meta human parts in the source file link so all id need to do is the IK's for the rest of the cloths they have and the male skeleton. tho import the body of skye first and have everything overlap that skeleton. Dont overlap the shirts, apparently their meant to be separate. When importing also make sure the T0 is ticked. but everything is in the source file link for you.

[OptimisticRaccoon]

Curious as to why you have to remove Virtual Bones on the ALS source Skeleton now, considering it worked before?
FWIW, I found a plugin that retargets without having to remove VBs https://github.com/wuguyannian/AnimAssistant
Also allows you to list and add VBs based on a JSON dataset.

{
    "BoneList":
    [
        {
            "BoneName":"VB Curves",
            "SourceBone":"root",
            "TargetBone":"root"
        },
        {
            "BoneName":"VB hand_r_to_hand_r",
            "SourceBone":"hand_r",
            "TargetBone":"hand_r"
        },
        {
            "BoneName":"VB ik_hand_l_no_gun_bone",
            "SourceBone":"VB hand_r_to_hand_r",
            "TargetBone":"Hand_L"
        },
        {
            "BoneName":"VB hand_l_to_hand_r",
            "SourceBone":"Hand_L",
            "TargetBone":"hand_r"
        },
        {
            "BoneName":"VB ik_hand_r_no_gun_bone",
            "SourceBone":"VB hand_l_to_hand_r",
            "TargetBone":"hand_r"
        },
        {
            "BoneName":"VB hand_l_to_ik_hand_gun",
            "SourceBone":"Hand_L",
            "TargetBone":"ik_hand_gun"
        },
        {
            "BoneName":"VB hand_r_to_ik_hand_gun",
            "SourceBone":"hand_r",
            "TargetBone":"ik_hand_gun"
        },
        {
            "BoneName":"VB ik_hand_r_with_gun_bone",
            "SourceBone":"VB hand_l_to_ik_hand_gun",
            "TargetBone":"hand_r"
        },
        {
            "BoneName":"VB ik_hand_l_with_gun_bone",
            "SourceBone":"VB hand_r_to_ik_hand_gun",
            "TargetBone":"Hand_L"
        }
    ]
}

The only downside is, it only copies Curves from AnimSequences, so the unused Curves for e.g. FootIk and Slots have to be added via a modifier or manually. At least I couldn't find a function to copy Curves from or add to AnimationBlueprints.
Maybe just create a dummy seq and add all Curves via a modifier.

Not really ALS-related, but plagues me since the dawn of time: Am I missing something, or is there no way around adjusting Sockets to fix the HeldObject Locations on retargeted Meshes?

[Sixze]

Unreal very often crashes during retargeting if there are virtual bones or sockets in the source skeleton, at least on 4.26 version. Therefore, it is safest to remove them from the source skeleton before retargeting.

In fact, you can attach objects to sockets, for this I even added following parameters: bUseGunBoneForOverlayObjects in BP_Als_Character and bUseGunBone in CR_Als, but the truth is, for standard animations from ALS, it is difficult to configure sockets so that the character always holds items correctly in all situations, even without retargeting.

But if you are going to make your own overlay animations and don't plan to use runtime retargeting for characters with different body proportions, then you can try to get by with just sockets.

[OptimisticRaccoon]

Soo, I can't really wrap my head around to how I set up the retargeting with Virtual Bones then.

Comparing the retargeted animation with the source looks pretty much identical, just the VB / IK rotations are completely off.

I did some experiments with CR and some Two Bone IK stuff with humbling results. I'm kind of confused about how it's meant to be done.

On a second note, shouldn't the IK Bones be unaffected by retargeting? 🤔

[Sixze]

Make sure the IK bones have the same rotation in the target skeleton as the bones they belong to (hands, feet).

[Sixze]

After spending some time experimenting with retargeting, I ended up with the following:

Skeleton

All skeletons must have the following IK bones:

• VB curves: root -> root

IK bones in the new skeleton should have the same tranform as the bones to which they belong, just like in the SKEL_Als skeleton.

Foot IK

Foot IK without IK bones

The following virtual bones are required:

• VB foot_l : VB curves -> foot_l
• VB foot_r : VB curves -> foot_r

In ABP_Als, the bUseFootIkBones parameter must be set to false.

In practice, this option works quite well, even though blending profiles and retargeting options are not available for virtual bones.

Foot IK with IK bones

No need for additional virtual bones.

In ABP_Als, the bUseFootIkBones parameter must be set to true.

After the retargeting has been done, you need to manually edit ALL of the new animations so that the foot IK bones have the same transform as the bones they belong to, otherwise, the movement animations may look very bad with foot IK enabled.

The advantage of this approach is that the individual foot IK bones allow you to customize blend profiles for them, and access to the retargeting options allows you to fine-tune the runtime retargeting so that different characters correctly place their feet on the ground, taking into account their body proportions.

Hand IK

After retargeting, the hands are likely to be incorrectly positioned relative to the hand IK bones and the character will not be able to hold the weapons correctly, so in any case, it is necessary do the following:

1. Take all animations where hand position matters (overlay, mantle, get up animations, etc.).
2. Manually edit the transform of the hands in these animations:
   • For weapon and object animations, the hand bones transform must be the same as the hand IK bones transform.
   • For mantle or get up animations, the character's hands need to be at ground level.
   • Etc.

It is also a good idea to edit other animations so that the hand IK bones have the same transform as the bones they belong to.

Hand IK without IK bones

The following virtual bones are required:

• VB hand_l_to_hand_r: hand_l -> hand_r
• VB hand_r: VB hand_l_to_hand_r -> hand_r

• VB hand_r_to_hand_r: hand_r -> hand_r
• VB hand_l: VB hand_r_to_hand_r -> hand_l

Also in ABP_Als, the bUseFootIkBones parameter must be set to false, and in BP_Als_Character, the bUseGunBoneForOverlayObjects parameter must also be set to false.

This option also works well, but only if the animations are made with the assumption that objects will be attached to sockets. Standard overlay animations from ALS follow this rule, with the exception of the M4 animations.

Hand IK with IK bones

The following virtual bones are required:

• VB hand_l_to_ik_hand_gun : hand_l -> ik_hand_gun
• VB ik_hand_r : VB hand_l_to_ik_hand_gun -> hand_r

• VB hand_r_to_ik_hand_gun : hand_r -> ik_hand_gun
• VB ik_hand_l : VB hand_r_to_ik_hand_gun -> hand_l

Also in ABP_Als, the bUseFootIkBones parameter must be set to true, and in BP_Als_Character, the bUseGunBoneForOverlayObjects parameter must also be set to true.

This is the preferred option because after retargeting, you still have to manually edit the resulting animations taking into account the tranform of the original hand IK bones.

Also, a separate gun bone, unlike sockets, can be animated, which can be very convenient in some cases, for example, if a character needs to move a weapon from his right hand to left.

[kevinbear91]

Foot IK with out IK bone there have a small error with v4.9 version in UE5.

VB ik_foot_l : VB curves -> foot_l
VB ik_foot_r : VB curves -> foot_r

VB ik_foot_l should be name to VB foot_l
VB ik_foot_r should be name to VB foot_r

In AlsConstants.h line 276 and line 282 there named virtual bones for foot IK without IK bones.

[PaulNguyen88]

Hi, I tried to retarget follow your reply but don't understand why everything is wrong.

My constraintbone is:
((ConstraintBone="root",ModifiedBone="ik_foot_root"),(ConstraintBone="foot_l",ModifiedBone="ik_foot_l"),(ConstraintBone="foot_r",ModifiedBone="ik_foot_r"),(ConstraintBone="root",ModifiedBone="ik_hand_root"),(ConstraintBone="hand_r",ModifiedBone="ik_hand_gun"),(ConstraintBone="hand_r",ModifiedBone="ik_hand_r"),(ConstraintBone="hand_l",ModifiedBone="ik_hand_l"),(ConstraintBone="ik_foot_l",ModifiedBone="VB foot_l"),(ConstraintBone="ik_foot_r",ModifiedBone="VB foot_r"),(ConstraintBone="ik_hand_gun",ModifiedBone="VB hand_r_to_ik_hand_gun"),(ConstraintBone="ik_hand_gun",ModifiedBone="VB hand_l_to_ik_hand_gun"),(ConstraintBone="hand_r",ModifiedBone="VB ik_hand_r"),(ConstraintBone="hand_l",ModifiedBone="VB ik_hand_l"),(ConstraintBone="ik_foot_root",ModifiedBone="VB foot_root"),(ConstraintBone="foot_l",ModifiedBone="VB foot_l"),(ConstraintBone="foot_r",ModifiedBone="VB foot_r"))
Is there something wrong here?

[AchimTuran]

Currently trying to retarget the UE5 Manny to ALS Refactored from commit 3970ae6.

These should be the right bone constraints:

((ConstraintBone="root",ModifiedBone="ik_foot_root"),(ConstraintBone="foot_l",ModifiedBone="ik_foot_l"),(ConstraintBone="foot_r",ModifiedBone="ik_foot_r"),(ConstraintBone="root",ModifiedBone="ik_hand_root"),(ConstraintBone="hand_r",ModifiedBone="ik_hand_gun"),(ConstraintBone="hand_r",ModifiedBone="ik_hand_r"),(ConstraintBone="hand_l",ModifiedBone="ik_hand_l"),(ConstraintBone="root",ModifiedBone="VB foot_root"),(ConstraintBone="foot_l",ModifiedBone="VB foot_l"),(ConstraintBone="foot_r",ModifiedBone="VB foot_r"),(ConstraintBone="hand_r",ModifiedBone="VB hand_l_to_hand_r"),(ConstraintBone="hand_r",ModifiedBone="VB hand_r"),(ConstraintBone="ik_hand_gun",ModifiedBone="VB hand_l_to_ik_hand_gun",),(ConstraintBone="hand_r",ModifiedBone="VB ik_hand_r"),(ConstraintBone="hand_r",ModifiedBone="VB hand_r_to_hand_r"),(ConstraintBone="hand_l",ModifiedBone="VB hand_l"),(ConstraintBone="ik_hand_gun",ModifiedBone="VB hand_r_to_ik_hand_gun"),(ConstraintBone="hand_l",ModifiedBone="VB ik_hand_l"))

[AchimTuran]

Yes super sad but the bone constraining does not work anymore after retargeting the animations. The solution I'm currently using is here, https://youtu.be/8HsJBHDuCU8.

Soon I will cover how we can implement a custom solver for UE5 IK retargeting to make the process of migration animations more stream lined.

In case you guys want to tinker with the approach you can do so within https://github.com/tuatec/TTToolbox/releases. ;-)

[Baelroak]

@SteveProteau I'm trying to get the foot ik retargets like you said you have done sucessfully, could you share your additions to the metahumans skeleton, did you add ik bones with the toolbox or just used virtual bones instead?

[SteveProteau]

@Baelroak I used virtual bones. However, if you can create ikbones from the toolkit they should work as well. Here is what I used:

I created using the instructions here (23 min into video):

https://www.bing.com/videos/search?q=ue5+ALS4+metahuman+foot+ik&&view=detail&mid=8F96E29652AF8C12FCF08F96E29652AF8C12FCF0&&FORM=VRDGAR&ru=%2Fvideos%2Fsearch%3Fq%3Due5%2BALS4%2Bmetahuman%2Bfoot%2Bik%26FORM%3DHDRSC3

[Baelroak]

@SteveProteau

Idk what I done differently it always turns out like this

Did you edit the control rig at all besides changing the skeleton hierarchy

[Baelroak]

well i got that part working, i think the bone constrain isn't working tho, not sure what I did wrong on that part

[IkuTri]

Following, @Baelroak were you able to get your bone constraint working?

[AchimTuran]

@PaulNguyen88
You are right, I did not test it with ALS refactored. There are so many forks, which is the main reason I made all tutorials with the base, the BP version. TTToolbox does not depend on any ALS version, users also use it with Lyra. It's a generic implementation that only needs the right configuration for the constraint bones. As long as the child and parent relationship is correctly setup it should work. 😉
If anyone is interested how TTToolbox works, I recommend watching the UE5 Manny series.