WebXR Raw Camera Access

[Maksims]

Since Quest 3 got released, it gets access to camera's raw textures.
We would want to have an API that gives an easy and integrated way to access and update such textures.
Specs: https://immersive-web.github.io/raw-camera-access/

EDIT:
Android has access to raw camera texture.

[Maksims]

Below is an example of an potential API. It implements access to views that session has.
Due to nature of WebXR API, when session starts views are not available straight away, and will become available on the first pose data availability. Also they might be removed and added during the session. So events should be used to detect when views are added.
It is designed to include depth information in this API in the future, and deprecate XrDepthSensing as it only assumed a single view information (mobile phone), but MR HMD devices now can provide multiple depth textures per view (two eyes).

// constants
// it comes from XRView eye property
XREYE_NONE = 'none' // usually it is a mobile screen
XREYE_LEFT = 'left'
XREYE_RIGHT = 'right'

// pc.XrManager
xr.views // XrViews object that provides access to views
xr.start(camera, pc.XRTYPE_AR, pc.XRSPACE_LOCALFLOOR, {
    cameraColor: true // new option to enable access to camera color information
});

// pc.XrViews
// this class represents number of views, their capabilities and access to individual views
views.supportedColor // true if color information is supported
views.availableColor // true if color information is available.
// It can be supported but not available due to lack of session eatures request or e.g.
// Quest 3 - implements necessary features, but does not expose `camera-access` to users.
views.get(eye) // returns XrView or null based on XREYE_*
views.list // an array of XrView instances
views.size // number of views
views.on('add', (XrView) => { }); // event when new XrView is added
views.on('remove', (XrView) => { }); // event when XrView is removed

// pc.XrView
// this class represents a specific view with its data
view.eye // returns a string (constants XREYE_*). 'none' - means it is not an eye, and
// most likely is a mobile screen
view.textureColor // returns a texture with color information. If color information is not
// available, texture will be blank 4x4

Example of how to request access to camera color:

app.xr.start(camera, pc.XRTYPE_AR, pc.XRSPACE_LOCALFLOOR, {
    cameraColor: true
});

Example of how to subscribe to access camera color:

const views = app.xr.views;

if (views.supportedColor) {
    // camera color might not be supported
    app.xr.on('start', () => {
        if (!views.availableColor) {
            // camera color might be unavailable
            return;
        }
        
        views.on('add', (view) => {
            // get access to color texture
            const texture = view.textureColor;
        });
    });
}

Example of how to access camera color with already running XR session and available views:

const views = app.xr.views;

if (views.supportedColor && views.availableColor) {
    const viewMobile = views.get(pc.XREYE_NONE);
    if (viewMobile) {
        const texture = viewMobile.textureColor;
    }
}

[Maksims]

I'm experiencing a challenge with the implementation, as Camera Access introduces "Opaque Texture" concept, which provides a WebGL texture that only lives for the duration of the requestAnimationFrame and should be unbound from the program (shader) at the end of animation frame.
So setting a provided opaque texture to the pc.Texture is a bit of a hack, but I do not know how within the engine to control removal of the texture from its bound programs.
Perhaps @mvaligursky would be able to help on that once got a spare time.

Also it is provided flipped vertically, I guess not much we can do about it without re-rendering?

[mvaligursky]

I need to do a similar solution to the video texture for WebGPU, which is valid only within the animation frame .. so perhaps we can find a single solution here. #5691

My current thinking was to do a copy to a normal texture to avoid this limitation. You could even do a flip during that time. Not ideal though.

[Maksims]

I need to do a similar solution to the video texture for WebGPU, which is valid only within the animation frame .. so perhaps we can find a single solution here. #5691

My current thinking was to do a copy to a normal texture to avoid this limitation. You could even do a flip during that time. Not ideal though.

Copying is probably a good first solution, as it will allow us implementing API, and if performance is an issue in the future, we can think of direct use optimisation.
What is the easiest way to copy texture? I haven't found a simple method to copy one texture into another in docs.

[mvaligursky]

At the moment probably just use drawQuadWithShader, that's probably the easiest.
To avoid per frame allocations, I typically extract the insides of it and keep those objects persistent.
But soon I'd hope to add a render pass for this, so you just schedule a render pass to make it easier. But not yet there.

[Maksims]

At the moment probably just use drawQuadWithShader, that's probably the easiest. To avoid per frame allocations, I typically extract the insides of it and keep those objects persistent. But soon I'd hope to add a render pass for this, so you just schedule a render pass to make it easier. But not yet there.

I've tried implementing it using framebuffer blit: https://github.com/playcanvas/engine/pull/5786/files#diff-4b7c531868727cdd27e21051721e99f838516832cf4d1ea1135c8719b8494d60R241-R295
Is this a viable option too?

[mvaligursky]

I commented on PR. I'd prefer we do not use GL directly in XR.