Add support for the OpenXR Eye gaze interaction extension

[BastiaanOlij]

With eye tracking becoming a more prevalent feature, it's time we add support for this.

This PR adds support for the core OpenXR eye gaze extension.

This is mostly untested at the moment:

• see @tom95 feedback below on this working on Quest Pro, we're still discussing how to include the permissions
• this likely also works on Pico 4 Pro but crashes on a normal Pico 4 due to an issue on Picos side.

This PR will remain in draft until the above are resolved.

In order to use this feature a new interaction profile needs to be added to the action map:

Than in your XR setup a new XRController3D node can be added like so:

This node will be positioned correctly and "point" in a direction where the user is looking. For HMD based eye tracking the location will be centred between the eyes of the user.

As an example a ray cast was added in the above screenshot, this could be used to detect the object the user is looking at.

Todos:

• Wait on improvements by @m4gr3d that allow control over feature flags and permissions from the loader plugins
• Add project setting to enable (opt-in) this extension
• Add checks in code to only enable extension if required permissions are given (this may require platform dependent code, need to investigate how we do this)

[dsnopek]

I don't know that any browser actually implements it, but in WebXR eye gaze is also represented as a controller-ish thing. Although, I don't think we'd currently expose it to Godot as a controller - it looks like it'd be an XRPositionalTracker using type TRACKER_UNKNOWN, but you could use WebXRInterface::get_input_source_target_ray_mode() to find out that it's eye gaze.

Does it make sense to add a new tracker type like TRACKER_GAZE or similar? Or, should I make WebXR set that to TRACKER_CONTROLLER as if it were a real controller?

I just want to make sure WebXR and OpenXR end up doing as similar things as possible :-)

[BastiaanOlij]

@dsnopek with openXR it just becomes part of the action system so we don't really indifferentiate between types. In that way the node being called XRController3D and the positional tracker having a controller type is kind of misleading, but seeing its the 99% use case it helps in people doing the normal stuff.

[BastiaanOlij]

Ok, just for future reference a bit of background information. This PR in theory is ready, but we're having some issues testing due to lack of capable devices. I'm working with Meta and Bytedance to see if we can get to a point we can prove this works correctly. As we're in feature freeze atm there is no rush, this is a 4.2 feature.

Anyway, the problem around this extension is that it's generally only available on "Pro" versions of headsets, i.e. Quest Pro, PICO 4 Pro, etc.

There is currently an ongoing discussion on how to deal with the scenario where an OpenXR runtime (the software written by the HMD vendor) supports an extension, but it is only available on some devices. So both Quest and PICO runtimes report this extension as available because the runtime implements it, but it will only work on the "Pro" versions of the respective headsets.

There is some debate on how the runtime should react on devices that do not support the feature. With this extension we use XrSystemEyeGazeInteractionPropertiesEXT to query whether the feature is supported but that is only half of the story. The question is whether supportsEyeGazeInteraction being false should result in filtering out the eye gaze interaction from the actionmap. Currently not doing so crashes Godot on some devices but this change is elaborate. The reading of the OpenXR spec however suggests that enabling an extension should make the interaction available to the game, and leave it up to the runtime to ignore this.

Off course we can't merge this PR until the above is resolved and headsets have been patched or Godot will stop working on effected headsets.

To be continued...

[m4gr3d]

Let's rename these two to explicitly refer to the interaction:

• XR_EYE_GAZE_INTERACTION_LIMITED
• XR_EYE_GAZE_INTERACTION_SUPPORTED

[BastiaanOlij]

I'm thinking of changing this all together. The specification language is a bit unclear and misleading.

From what I understand the extra structure we can populate during session creation determines whether gaze interaction is available, not whether its full features.

Basically the extension being available means the runtime supports it, the struct queries whether the device in use actually supports it. So for instance, the Meta XR runtime supports eye tracking so the extension will always be available, but only the Quest Pro actually has an eye tracker so only that will return that the gaze is actually supported.

So the outcome of this should be a simple Yes/No, we can use the eye gaze interaction if the extension is available AND we get true on the supported flag.

[tom95]

Hi, we've recently tried out this PR with the Quest Pro. The following steps needed to be taken to make it work on our end:

1. Add gaze tracking permission to the AndroidManifest.xml [1] as documented here [3].
2. Extend GodotApp.java to ask for gaze tracking permission [2]
3. Follow the setup as @BastiaanOlij described in the above comment. (Likely unrelated: we had some troubles configuring a custom action map and ended up copying the map from openxr-tools and adding eye tracking).

Then, eye tracking worked flawlessly. (cc @BennytheBomb)

[1]

<uses-feature android:name="oculus.software.eye_tracking" android:required="true" />
<uses-permission android:name="com.oculus.permission.EYE_TRACKING" />

[2]

package com.godot.game;

import org.godotengine.godot.FullScreenGodotApp;
import android.content.pm.PackageManager;

import android.os.Bundle;

/**
 
Template activity for Godot Android builds.
Feel free to extend and modify this class for your custom logic.
*/
public class GodotApp extends FullScreenGodotApp {
    private static final String PERMISSION_EYE_TRACKING = "com.oculus.permission.EYE_TRACKING";
      private static final int REQUEST_CODE_PERMISSION_EYE_TRACKING = 1;

    @Override
    public void onCreate(Bundle savedInstanceState) {
        setTheme(R.style.GodotAppMainTheme);
        super.onCreate(savedInstanceState);
        requestEyeTrackingPermissionIfNeeded();
    }

    private void requestEyeTrackingPermissionIfNeeded() {
        if (checkSelfPermission(PERMISSION_EYE_TRACKING) != PackageManager.PERMISSION_GRANTED) {
            requestPermissions(new String[] {PERMISSION_EYE_TRACKING}, REQUEST_CODE_PERMISSION_EYE_TRACKING);
        }
    }
}

[3] https://developer.oculus.com/documentation/native/android/move-eye-tracking/

[BastiaanOlij]

Thanks @tom95 , amazing to hear its working on an actual device. We're having some issues with PICO atm that will prevent merging until they fix an issue on their end (or we'll break PICO support).

@m4gr3d we'll need to look into how we're going to embed the permission, and look further into what permissions other vendors come up with. Seeing this is a core extension this should really be a core permission.

[BastiaanOlij]

Feedback from PICO is that we need to add <uses-permission android:name="com.picovr.permission.EYE_TRACKING"/> to the manifest.

[m4gr3d]

Feedback from PICO is that we need to add <uses-permission android:name="com.picovr.permission.EYE_TRACKING"/> to the manifest.

@BastiaanOlij, should this permission be added to the plugin refactor?

[BastiaanOlij]

@BastiaanOlij, should this permission be added to the plugin refactor?

Yeah I think thats worth doing, still need to think about how we're going to implement asking for the correct permission and checking in the module whether permission has been given

[m4gr3d]

@BastiaanOlij, should this permission be added to the plugin refactor?

Yeah I think thats worth doing, still need to think about how we're going to implement asking for the correct permission and checking in the module whether permission has been given

That's where the feature tags come into play. There are two scenarios:

• The extension only requires a permission / entry in the manifest; in this case, the export plug-in options are used both to select and insert the permission and to include the corresponding feature tags using https://docs.godotengine.org/en/stable/classes/class_editorexportplugin.html#class-editorexportplugin-method-get-export-features
• The extension requires the above + runtime logic for user confirmation; similar to the above the necessary permissions are added via the plugin options. Here however the feature tags are added by the plugin logic at runtime after receiving user confirmation (or any other necessary runtime logic).

In both scenarios the extension is gated behind the feature tag check using custom features and OS.has_feature(name).

We may need to add logic to so the plugin can request extensions to be enabled on demand after startup.

[BastiaanOlij]

@m4gr3d implemented the missing bits in #82614 (namely the support for checking permissions). This PR should be closed once that PR is merged.

[akien-mga]

Superseded by #82614.