[Enhancement][Android] Add support for MV-HEVC

[amedinae]

Android virtual reality headsets, such as the Pico 4 ultra, also support the MV-HEVC format

[konradrvl]

As far as I know the XR2Gen2 Chipset the Pico 4 ultra uses does not support MV-HEVC video. It shares this with the Quest 3 (that also uses the XR2Gen2). I had a look at the codecs supported by the Quest 3 XR2Gen2 and it does support HEVC but sadly without the Multiview extension.

However that doesn't stop PICO from claiming their device is compatible with the MV-HEVC format on their website. What they mean by this is that if you upload a MV-HEVC video through their gallery app on your phone they convert it to a side-by-side video internally to be played on the headset. If you load the MV-HEVC file directly on the device however it will not work.

I am not sure if it is possible to add support for MV-HEVC through a firmware update of the XR2Gen2 or if hardware changes are needed, but for now neither the Pico 4 ultra nor the Quest 3 support MV-HEVC video.

[Chris-RH]

Ahh, thanks for that information @konradrvl, that's very helpful.
I suppose its something that we can look to support when devices (and Exoplayer) are able to support it natively.

[imetandy]

A note here that we have successfully tested MV-HEVC natively on the Quest 2 and the Quest 3 and it plays back well.
The files were loaded onto the headset via ADB, so there does seem to be support here for this.

We're trying to distribute a film which required quite a high bitrate, which is why we opted to explore MV-HEVC which is giving us amazing quality and smooth playback.

We have not yet tested with AV Pro. To confirm, are we waiting for Exoplayer to catch up before this will likely work, or should this be pretty plug and play as of now?

Thanks for all the amazing work, this plugin definitely saved a huge amount of dev pain on our side! :D

[konradrvl]

Hey @imetandy

That would be amazing news!

I just now looked into the decoder profile for the HEVC decoder on the Quest 3 again and found no profile mentioning MV-HEVC:

HEVCProfileMain (0x1): HEVCHighTierLevel62 (0x2000000)
HEVCProfileMainStill (0x4): HEVCHighTierLevel62 (0x2000000)
HEVCProfileMain10 (0x2): HEVCHighTierLevel62 (0x2000000)
HEVCProfileMain10HDR10 (0x1000): HEVCHighTierLevel62 (0x2000000)
HEVCProfileMain10HDR10Plus (0x2000): HEVCHighTierLevel62 (0x2000000)

Here is the full decoder profile: quest3decoder.txt

Just some clarifying questions to understand what you did:
Were you using the Quest 3s native hardware decoder or your own software based solution?
Are you playing stereoscopic videos?
Did you use AVfoundation to create the MV-HEVC video?

Would be awesome if this works!

[imetandy]

Hey @konradrvl !

Rough workflow below:

1. Create a 360 animation png sequence, stereoscopic. (Settings are 5760x5760px, over/under stereoscopic 3D)
2. Convert sequence to master ProRes 4444 file with ffmpeg
3. Use Spatial cmd tool on MacOS by Mike Swanson (This utilizes AVfoundation to create the MV-HEVC encoded file, very well documented) - Link to his blog
   I used the following one-liner:
   ./spatial make -i /Volumes/docs/theInputFile.mov -f ou --cdist 30.0 --hfov 360.0 --hadjust 0.2 --projection equirect --hero right --primary right -o /Volumes/docs/output.mov
4. Copy file to the movies folder on the headset (have done this via ADB, and by dragging and dropping the file into the folder on windows)
5. Use the inbuilt file-viewer app to watch the video. This reads as being played back as a VR 360 file in the headset, but ultimately is stereoscopic 3D due to the offset in each of the two flat video files in the MOV container, certainly looks 3D to my eyes.

Hope this helps! Feel free to DM or email me at - andrew (at) marshmallowlaserfeast (dot) com - happy to share over a test file if you want to test it out on your end.

[benb0jangles]

As far as I know the XR2Gen2 Chipset the Pico 4 ultra uses does not support MV-HEVC video. It shares this with the Quest 3 (that also uses the XR2Gen2). I had a look at the codecs supported by the Quest 3 XR2Gen2 and it does support HEVC but sadly without the Multiview extension.

However that doesn't stop PICO from claiming their device is compatible with the MV-HEVC format on their website. What they mean by this is that if you upload a MV-HEVC video through their gallery app on your phone they convert it to a side-by-side video internally to be played on the headset. If you load the MV-HEVC file directly on the device however it will not work.

I am not sure if it is possible to add support for MV-HEVC through a firmware update of the XR2Gen2 or if hardware changes are needed, but for now neither the Pico 4 ultra nor the Quest 3 support MV-HEVC video.

I have a demo unit of the pico 4 ultra enterprise and it is able to record spatial video & play it back. But if I copy the file to my quest 3s it plays as flat video. I read that the quest app 4XVR plays MV-HEVC but i tried it and not successful. The video files I have recorded are in MV-HEVC format if I check file information.

[waqia]

As far as I know the XR2Gen2 Chipset the Pico 4 ultra uses does not support MV-HEVC video. It shares this with the Quest 3 (that also uses the XR2Gen2). I had a look at the codecs supported by the Quest 3 XR2Gen2 and it does support HEVC but sadly without the Multiview extension.

However that doesn't stop PICO from claiming their device is compatible with the MV-HEVC format on their website. What they mean by this is that if you upload a MV-HEVC video through their gallery app on your phone they convert it to a side-by-side video internally to be played on the headset. If you load the MV-HEVC file directly on the device however it will not work.

I am not sure if it is possible to add support for MV-HEVC through a firmware update of the XR2Gen2 or if hardware changes are needed, but for now neither the Pico 4 ultra nor the Quest 3 support MV-HEVC video.

Pico4 ultra DO support MV-HEVC video natively， including recording and display. I knew some developers of Pico 4 ultra media team, they confirmed that they cooperated with Qualcomm team and finish the hardware encoder and decoder for MV-HEVC video

[pdavies011010]

I would also love to see AVPro support MV-HEVC, on whatever platforms it can be supported!