XR_ERROR_RUNTIME_FAILURE: "Simultaneous hands and controllers cannot be enabled while Body API is active"

[BattleAxeVR]

Hello, when I call xrResumeSimultaneousHandsAndControllersTrackingMETA in my IGL-based PC VR streaming app, in order to enable multi-modal, I get an error:

Specific Line of code:

https://github.com/BattleAxeVR/igl/blob/02011a8d3c311d4e842b70376eb8aa1e758b3951/shell/openxr/mobile/XrApp.cpp#L2234

I originally logged the issue here, but was asked to recreate it here in IGL to get some visibility inside Facebook and hopefully a solution can be found.

Repro steps:

1. Enable Meta Body Tracking API via OpenXR
2. Enable Hand Tracking EXT
3. Enable multi-modal hands/controllers, that depends on 2 but is mutually exclusive with 1.

The order of 1-2 above can be swapped but leads to body tracking failing with the same XR_ERROR_RUNTIME_FAILURE, first Hand tracking EXT, then Multi-modal, then Body tracking fails.

Why do I need this?

Well, full body tracking is useful generally for immersion, presence, and fun, but specifically for my app I need the waist pose to implement waist-oriented locomotion, which reduces if not eliminates VR simulation sickness caused by rotations induced by head-oriented during smooth movement, which couples the head and body into one. Waist-loco decouples head and body so you keep walking straight = no rotation, no sickness (I highly encourage people to try it to confirm it for themselves, it works in any SteamVR game using my OK Cloud Streamer app)

The multi-modal support is also needed for my game, to use 2-handed weapons, or 1 main hand weapon and a free offhand, to do things like open doors, turn keys, pick up / manipulate objects, pull levers, cast spells using complex gestures involving fingers, etc. The possibilities are endless. I need at least 1 VR controller with a thumbstick + waist-oriented locomotion in order to navigate in-game. I cannot ship one without the other and permit these combinations of activities.

I don't see any technical reason why FBT and HT EXT can't be made to work at the same time, after all, the hand is a part of the body, and HT EXT returns a subset of the joints that FBT does. FBT + multi-modal should therefore be possible.

I hope Facebook will fix this (if you aren't already doing so) so I can use all these features at the same time to achieve my vision. Thanks.

[BattleAxeVR]

@corporateshark Any updates?

By the way, there's another bug in Meta's OpenXR runtime returning incorrect status codes, this time when enabling fidelity tracking, which returns XR_SUCCESS on Quest 2 and Pro whereas the feature only actually works on Quest 3. I can't even add up how much time this has wasted.

If someone could add it to your internal bugs that would be greatly appreciated. Last time I reported something like this on Meta forums it took six months to get fixed and that was only after I got Mr. Carmack's help to elevate it (sadly he's gone now).

https://x.com/BattleAxeVR/status/1811136289359724985

[rokuz]

Hey @BattleAxeVR ! Thanks for reminder! This problem is not directly related to IGL, we can deliver bugs to the team that is up to implement OpenXR. However, unfortunately we can't affect on their roadmap or priorities.

[BattleAxeVR]

I will gladly make a couple fresh bug reports in an appropriate forum or github repo, but the last time I tried doing so in Oculus Dev forums (the OpenXR runtime on Quest 2 on PC only ran at 67 FPS due to a change in Windows 10 default timings, and nobody seemed to care! Including the OpenXR SDK Source repo maintainers who asserted the Oculus XR runtime was "compliant" as if that means it covers performance issues). It took six months to even be acknowledged and I had to beg Carmack for help to escalate it. There has to be a better way. These issues are holding back my game development.

[BattleAxeVR]

I reported the issue in the Oculus Dev forums, @corporateshark @rokuz can I ask you guys a favour and get someone to look into it?

https://communityforums.atmeta.com/t5/OpenXR-Development/quot-Simultaneous-hands-and-controllers-cannot-be-enabled-while/m-p/1224669#M908

[BattleAxeVR]

Any update? I can't deliver my Stylus app without this bug fixed.

[rokuz]

Hey @BattleAxeVR ! Sorry to hear that. Unfortunately, no updates. Relevant team is aware of it. As I mentioned earlier, IGL's team can't affect on them in terms of priorities.