Move video playback out of core and into an officially supported GDExtension

[Calinou]

Note: This change was discussed with reduz and others and is probably good to implement.

Describe the project you are working on

The Godot editor 🙂

Describe the problem or limitation you are having in your project

Video playback in Godot currently leaves a lot to be desired:

• It has a lot of bugs (see open bug issues for Theora, WebM).
• There are performance issues due to not being able to benefit from hardware video decoding, especially on mobile hardware.
• It takes valuable space in the binary due to video decoding libraries being large. (libtheora and libvpx also have their own dependencies that could be removed from core, such as libopus.)
• It lacks features that are a deal-breaker in some projects such as seeking.
• The number of supported formats is fairly limited, with patent-encumbered formats not being supported.
• Library updates are difficult for maintainers to perform. In general, libvpx (used for VP8 and VP9 decoding) isn't exactly known to be easy to work with.
• It's difficult to compile on unconventional platforms, making the build process unnecessarily complicated there.

We have very few contributors knowledgeable with video decoding libraries, so bug fixes and improvements are rarely seen nowadays.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

With GDExtension (the replacement of GDNative in 4.0), we can move video decoding to an officially supported add-on. This add-on will likely use FFmpeg like godot-videodecoder currently does, but it may also use another library depending on code size, maintenance quality and licensing.

There are many benefits to moving video playback out of core:

• The binary size penalty is removed from the Godot editor and export template binaries. Instead, the size cost is moved to the add-on's compiled libraries.
• Hardware video decoding could be used on compatible hardware and software.
• Support for patent-encumbered formats can be exposed optionally. (If this is implemented, this will be disabled by default.)
  • Users will need to check their country's regulations and possibly acquire licenses, especially for commercial use. It is possible for open source software to support patent-encumbered formats – otherwise, VLC would not be able to play back .mp4 videos 🙂

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

Perform a change like godotengine/godot#52003, but for VideoPlayer.

If this enhancement will not be used often, can it be worked around with a few lines of script?

No.

Is there a reason why this should be core and not an add-on in the asset library?

Video decoding needs to have hooks in the engine to be efficiently implemented, so it needs dedicated GDExtension work.

[fire]

I'll contact anyone still on the video decoder team at https://github.com/kidrigger/godot-videodecoder.

@kidrigger @jamie-pate

[jamie-pate]

For the video extension, i just noticed that one of my devs is running Apple silicon but we didn't notice the gdnative dylib failed to load because it falls back to the built in playback (graceful degradation since it doesn't support stuff like steam position), so it would be best to support building for all platforms when the built in video player support is removed.

We also have no build support currently for web or mobile.

To do this properly we need yuv texture support because that's a how all the hw accelerated decoders work.

I've seen shaders that do it so maybe it just needs to be an option on the spatial material

[kidrigger]

I'm working on the GDExtension support now. I'll add a new proposal but here's the gist of it.

VideoStream and VideoStreamPlayback classes need work to be extended from GDExtension.
There are two options

1. Keep our playback and let plugins add decoders
   • This is the way GDNative plugin currently works.
   • This will be faster to implement as we just need the old Decoder Interface to be updated for GDExtension.
2. Make the playback classes GDExtension capable and let the Extension implementation set their own.
   • This option adds flexibility for the extension developers
   • Reduces the playback logic on Godot core
   • Bypasses the performance constraints caused due to design of the GDNative implementation. Will improve performance by allowing the implementation to decide playback logic
   • This will take longer as the new Playback extensions will need to be developed

I have started work on Option 2 unless Option 1 is asked for due to time constraints.

[Catchawink]

@kidrigger any updates on this or ways I could possibly help? I'm looking to address #2553 and it appears to be contingent on this one.

[kidrigger]

@Catchawink godotengine/godot#62737
The PR needs to be merged.
The WebM plugin is working pretty well otherwise.

Resolving conflicts right now: ETA < 1 hr.
All conflicts have been resolved.

Who should I asked to review this quickly?

[lostminds]

As finding a decoder library with suitable features, license and platform support seems difficult, could an alternative be to instead rely on platform specific built in system services for this? At least for MP4 / H264 there seems to be support for this in Windows via Microsoft Media Foundation, macOS/iOS via CoreVideo, Android via MediaCodec and possibly in some linux versions as well?

The tradeoff would of course be that more platform specific code would be needed, and feature support will vary. And for web deployments for example this might not be possible. But even a lowest common denominator feature set will likely be a big improvement over the current support, and while not ideal I'm sure it won't be the only feature that is not available on all platforms. One issue though is that the currently only supported format OGV/Theora is likely not supported by any of these system services, since it's so unusual. So moving exclusively to using system decoders would break compatibility with existing projects using these videos.

[voylin]

@Calinou I have successfully made a GDExtension which has video and audio playback with seeking. Not certain if this could be turned into an official extension though. I know a lot more work would be necessary to make it more user friendly. It uses FFmpeg

If I were to be allowed to work on an official gdextension, how would the process be? I'd probably also look into #8049 to see if it's possible for me to also implement that as I both need it for my video editor and as it could improve performance for people using the GDExtension.

[voylin]

A little update on this, I got a GDExtension working for creating video playback. It is not a drop in replacement as you still need to write the code yourself for displaying the frames in order. But it is a beginning. https://github.com/VoylinsGamedevJourney/gde_gozen

Only question would be is if I can turn this into an officially supported GDExtension as I don't know how it can become "officially supported" ^^" So if you think I'm on the right track, let me know and I can try to create a Player node ;)

[Calinou]

Great work 🙂

Only question would be is if I can turn this into an officially supported GDExtension

I think the extension would need to be production-ready first, which means it should be able to work as a drop-in replacement for the existing built-in video playback system (other than reencoding your videos to the desired format).

[voylin]

I think the extension would need to be production-ready first, which means it should be able to work as a drop-in replacement for the existing built-in video playback system (other than reencoding your videos to the desired format).

If it were to use FFmpeg as a library, is this okay for an official GDExtension?

[kobligo]

From https://www.ffmpeg.org/legal.html

FFmpeg is licensed under the GNU Lesser General Public License (LGPL) version 2.1 or later. However, FFmpeg incorporates several optional parts and optimizations that are covered by the GNU General Public License (GPL) version 2 or later. If those parts get used the GPL applies to all of FFmpeg.

I know nothing about licenses. Is this compatible or not?
Isn't FFMPEG Godot's only real option when it comes to creating a proper video player?

[DeeJayLSP]

I know nothing about licenses. Is this compatible or not? Isn't FFMPEG Godot's only real option when it comes to creating a proper video player?

In order to play a video, you have to:

1. Demux the container
2. Decode the video stream with the correct decoder
3. Same as above but for the audio stream

FFmpeg is a fairly common implementation that mostly makes the 3 things above easy. A video player could be done by directly implementing the decoders and demuxers into a VideoStream class, but doing so would require a bit more work.

[fire]

I would prefer to keep one video codec in the core of godot engine for testing purposes.

[Calinou]

I know nothing about licenses. Is this compatible or not?

It is compatible with Godot's license, but LGPL imposes restrictions on dynamic linking. This is why it can't be integrated in core and should be an add-on instead, as not everyone can deal with these restrictions (e.g. iOS exports due to static linking being mostly nonexistent there).

The GPL-licensed portions should be disabled at compile-time, but most library-based usage of FFmpeg does that already.

[jamie-pate]

I would prefer to keep one video codec in the core of godot engine for testing purposes.

I agree that directly integrating the simplest, most compatible* codec/small library is a good idea for broad compatibility and a reference implementation in the core engine, even if we are stuck with software decoding of a single video format**

* Compatible with all platforms and our license, not your content encoding. If a developer really really needs video to work everywhere they can re-encode it to fit the engine/performance constraints! Re-encoding is a much simpler solution than adding broad support for any kind of video across all platforms.

** Theora (VP3-derived codec) video kind of sucks these days, swapping out libtheora completely to libvpx (VP8/VP9) would be wonderful if someone wants to take that on ;D

Ideal solution (long term)

It would be nice to have platform integration when available though, for devs who are targeting specific platforms. These seem like reasonable targets to me for an extension which could be migrated into core at a later date if the implementation is smooth enough.

Vulkan does video decoding

Vulkan now has an extension for video decoding!
https://github.com/KhronosGroup/Vulkan-Docs/blob/main/proposals/VK_KHR_video_queue.adoc#3-proposal
This seems like a great direction to shoot for long term with our API. I don't think Godot 4.3's VideoStream fills the requirements for this to work.

Design-wise the features of their extension would probably be well thought out and offer maximum performance + utility, VideoStream should probably evolve in this direction.

Windows/Linux drivers for AMD/NVIDIA/Intel all support the extension
mpv-player/mpv#13909 has more research on desktop compatibility (as of May 29, 2023)

How FFMPEG uses it

Other Platforms

Mac/iOS/Android would have to have an alternate implementation. Preferably one that can hardware decode directly to video memory.

Android 15:

MUST NOT enumerate support for the VK_KHR_video_queue, VK_KHR_video_decode_queue, or VK_KHR_video_encode_queue extensions.

(probably because they want you to use https://developer.android.com/reference/android/graphics/SurfaceTexture or VK_ANDROID_external_memory_android_hardware_buffer: https://stackoverflow.com/a/43550804/193232)

Metal seems like it has similar capability

https://www.khronos.org/blog/khronos-finalizes-vulkan-video-extensions-for-accelerated-h.264-and-h.265-encode

PS

FFMPEG seems like a good idea because it has broad support for codecs/containers and it's a strong brand but as a game engine, 'everything and the kitchen sink' isn't really a good approach.

🤔 this lib does cool video stuff on top of FFMPEG (just leaving this here as a code reference, it's LGPL too)

[DeeJayLSP]

I wonder if MPEG-1 would be a good choice. Compared to Theora, there is a very simple and lightweight library for it, decoding is significantly faster in single thread, and quality is just slightly worse (given you know how to use the encoding parameters).

Out of curiosity, I went to experiment with a few encoding parameters and resulted in me writing this article with the results.

[jamie-pate]

My experiments led to the conclusion that vp9 software decoding is efficient enough for desktop use.

It can produce reasonable quality at very low bitrates (like, 512KiB @1080p)

With the current approach in godot, a lot of cpu time is actually spent converting the decoded pixels from yuv to rgb anyways. The video resolution plays a major role. (I think it took more time than decoding!)

Theora is already very far behind on the quality/bitrate curve, i only suggest replacing it because vp9 is so very far ahead (equivalent of h265)

Can you compare the output with your mpeg1 tests?

I use this ffmpeg script:

RATE=${3:-0.5M}
QUALITY=${4:-23}
INPUT="$1"
LOGFILE="$1-pass"
OUTPUT="${2:-${1%.*}.webm}"

ffmpeg -i "$INPUT" -c:v libvpx-vp9 -b:v $RATE -pass 1 -passlogfile "$LOGFILE" \
           -an -f webm -y \
           -threads 12 \
           -tile-columns 6 -frame-parallel 1 -row-mt 1 \
           -crf $QUALITY \
           /dev/null
ffmpeg -i "$INPUT" -c:v libvpx-vp9 -b:v $RATE -pass 2 -passlogfile "$LOGFILE" \
           -c:a libopus -threads 12 \
           -tile-columns 6 -frame-parallel 1 -row-mt 1 \
           -auto-alt-ref 1 -lag-in-frames 25 \
           -crf $QUALITY -y \
           "$OUTPUT"

[DeeJayLSP]

Can you compare the output with your mpeg1 tests?

Quality-wise, VP9 was obviously the best.

But since I was talking about decoding speed, I downloaded the video used in the article at 1440p, then encoded it to MPEG-1, Theora, VP8 and VP9 to test the decoding speeds.

The commands used:

# You have to specify libvpx(-vp9) or else FFmpeg will use the built-in, non-permissive ffvp8/ffvp9 decoders.
# Also use -threads 1 before the input, to measure their single-thread speed only.
ffmpeg -threads 1 -c:v libvpx -i vp8.webm -f null /dev/null
ffmpeg -threads 1 -c:v libvpx-vp9 -i vp9.webm -f null /dev/null
ffmpeg -threads 1 -i theora.ogv -f null /dev/null
ffmpeg -threads 1 -i mpeg1.mpg -f null /dev/null

The resulting speeds in my CPU:

• VP9: 2.85x
• VP8: 4.05x
• Theora: 5.24x
• VP9 (without -threads 1): 10.8x
• MPEG-1: 12.7x
• Theora (without -threads 1): 24.7x
• MPEG-1 (without -threads 1): 37.5x

[Calinou]

Remember that we had VP8/VP9 decode in 3.x and had to remove it because it was buggy in practice. simplewebm/libvpx are much more complex to integrate than libogg/libtheora and they require you to use their custom assembler (yasm) to achieve good performance.

I don't think the buildsystem situation around WebM libraries has changed much since 2020, especially since most of the development focus from Google is on AV1 now.

Regarding MPEG-1, Theora is significantly more efficient (and it's already considered not efficient enough by a lot of users), so I wouldn't go that route.

[DeeJayLSP]

simplewebm/libvpx are much more complex to integrate than libogg/libtheora

Theora isn't free of bugs either.

In fact, it seems libtheora itself is quite buggy. I recall someone converting a Theora video to WebM in a certain Godot issue, and it seems the problem carried over. I even have some problems playing a video on the mpv player.

Still in defense of MPEG-1, I have encoded a reference video into Theora and MPEG-1 using the best encoding settings I've spent hours trying to achieve, while trying to maintain a close size to the original (encoded in 1440p then downscaled):

https://www.youtube.com/watch?v=hBK_2V3Or70

(I swear I didn't do anything to break Theora)

Additionally, like I mentioned, there is pl_mpeg, which is a single header MPEG-1 implementation with SDL and GLEW player examples, and I bet it would be much easier to maintain than Theora.

Regarding MPEG-1, Theora is significantly more efficient

So, I genuinely disagree with the "significantly" part.

[jamie-pate]

It sounds like libvpx is pie in the sky since it was already removed for the same reasons.

I re-reviewed the core criteria of this proposal, and if the choice is:
a. removing video completely
b. switching to MPEG-1

Then I'd agree with MPEG-1 here. A single header MPEG-1 implementation with an MIT license sounds like a slam dunk to solve these points without making things worse. Remember that when theora was added MPEG-1 would have still been license/patent encumbered. It's been hard for me to shed that bias!

How many of these still apply to libtheora vs libvpx (since libvpx is already gone)

• It lacks features that are a deal-breaker in some projects such as seeking.
• libvpx: Update to version 1.7.0 godot#8334 In general, libvpx (used for VP8 and VP9 decoding) isn't exactly known to be easy to work with.
• It's Build failure for libvpx on ARMv8 with 32-bit Linux OS: Inlining failed in call to NEON instructions godot#30914, making the build process unnecessarily complicated there.
• It takes valuable space in the binary due to video decoding libraries being large. (libtheora and libvpx also have their own dependencies that could be removed from core, such as libopus.)

(i love .ogg, please don't remove libopus :D )

It also meets @fire 's request:

I would prefer to keep one video codec in the core of godot engine for testing purposes.

An officially supported extension for platform-level decoding of advanced formats would support the other original criteria:

• The number of supported formats is fairly limited, with patent-encumbered formats not being supported.
• There are performance issues due to not being able to benefit from hardware video decoding, especially on mobile hardware.

But since I was talking about decoding speed

Looks like MPEG-1 is 4x faster, but VP9 gives 5x bitrate reduction with better quality. IMO that still a win for VP9! Unfortunately this is irrelevant if VP9 is off the table :D

A large portion of video playback frame time in godot is the colorspace conversion. Any benchmarks need to be run with the current VideoStream workflow to include that number. (increasing quadratically with resolution)

I'm really really curious how much the colorspace conversion would cost vs the decoding speed of MPEG-1

March 2, 2022: I profiled software decoding and the cpu was dominated by yuv->rgb (* on the main game thread) because in the ffmpeg plugin the decoding happens in a separate thread

You maybe can estimate this cost by adding an ffmpeg filter to convert the frames to RGB before sending them to /dev/null with -threads 1 and compare with the current benchmark?

[fire]

I think the best we can do is use mpeg-1 and replace the code for libtheora with stubs that do nothing.

There's a rgba to yuva shader done by @EIREXE

EIRTeam/EIRTeam.FFmpeg@49efa2a

[Calinou]

I think the best we can do is use mpeg-1 and replace the code for libtheora with stubs that do nothing.

I'd say removing Theora support in 4.x without providing it as an add-on is out of the question, as it would not only break compatibility, but also require users to convert their existing videos to another lossy format (and therefore suffer from double compression).

[jamie-pate]

I'd say removing Theora support in 4.x without providing it as an add-on is out of the question

Deprecate it for now when MPEG-1 is added? I don't think anyone's lining up to push VideoStream forward at the moment, so adding MPEG-1 would probably wait until someone's ready to do that work. It would probably be a good way to learn the systems involved also.

Personally I've never had good experiences with Theora encoding and it's left a bad taste in my mouth when I've used it professionally. Neither MPEG-1 or Theora are going to be good for anything over 720p, but with proper docs we can steer users to accept/embrace the limitations.

There's a rgba to yuva shader done by @EIREXE

(this is part of the 'push forward' VideoStream needs, YUV support. I'm sure there's other features that are missing though)

[DeeJayLSP]

If anyone is interested, I made an attempt at implementing MPEG-1 support through pl_mpeg. Almost nothing works, but at least you can set a .mpg file into VideoStreamPlayer without crashing.

The only thing that is working 100% is probably demuxing (it detects width*height correctly).

[lostminds]

Having struggled a bit with using the currently implemented video features for a number of demo/tutorial video snippets in a project I can perhaps offer a "user" perspective on this. I would support the original idea of moving to something like ffmpeg (or similar), even if this due to legal reasons requires it to be a separate GDExtension. And with that just leave theora support as is for compatibility or remove it with deprecations warnings if the new external solution could also handle decoding and displaying theora.

The main issue as I see it with the current implementation is simply that the supported ogv/theora format is very unusual, which requires extra specific tools and steps in the development workflow to re-encode videos. To make the workflow less cumbersome and reduce the initial threshold to use these features I'd say the most important change would be to directly support common modern formats, at least for import (like mp4 containers with H.264 encoding).

The second issue I've had with the current implementation is the very limited display and interaction support, for example you need to start playing a video to get something to show in a VideoStreamPlayer (see godotengine/godot#92050) and there's no support for searching. This severely limits how you can use videos in UIs and make them hard to work with in the editor. These issues are I think not directly dependent on encoding, but would likely be easier to fix with some external library like ffmpeg.

Third, I would also consider future possibilities of not only decoding/displaying video but also encoding video that integrating something like ffmpeg could offer. While of course displaying video will be more common, encoding video to modern formats would bring new opportunities. Both for editor/development workflows to render video assets and video "filter" style features in Godot projects using the rendering engine combined with importing and/or exporting common video formats.

[jamie-pate]

I would support the original idea of moving to something like ffmpeg (or similar), even if this due to legal reasons requires it to be a separate GDExtension.

This unofficial extension exists for godot 4 (only has builds for windows64/linux64)
https://github.com/EIRTeam/EIRTeam.FFmpeg (maybe someone can add this link to the description? It's the spiritual successor to https://github.com/kidrigger/godot-videodecoder)

[fire]

@DeeJayLSP

Error: .\thirdparty/misc/pl_mpeg.h(1390): error C2220: the following warning is treated as an error
Warning: .\thirdparty/misc/pl_mpeg.h(1390): warning C4996: 'fopen': This function or variable may be unsafe. Consider using fopen_s instead. To disable deprecation, use _CRT_SECURE_NO_WARNINGS. See online help for details.

Will this require much code tinkering?

[DeeJayLSP]

Will this require much code tinkering?

Would be much easier if there was a *_NO_STDIO define like most single-header libraries. That function isn't even used by my implementation, so I just patched it out.

Audio is barely working now. I have no idea of how to use mix_callback correctly. Video remains not showing. Both are working now, just not optimally.

[fire]

I have my (not working) attempt at porting to 4.0 Windows Windows Media Foundation. https://github.com/V-Sekai/godot/tree/archived/wmf

Which is based on working Godot Engine 3. https://github.com/V-Sekai/godot/tree/vsk-wmf-3.5

Edit:

The approach is we use the native Operating System decoders and (encoders). We know that gstreamer exists for Linux, WMF exists for Windows. There's something for MacOS and Android.

[DeeJayLSP]

My pl_mpeg implementation is almost done. Playback seems to support both video and audio correctly.

There are two problems, however:

• pl_mpeg's built-in YUV to RGB conversion seems extremely slow. I couldn't figure out a way to link it to yuv2rgb (no strides unlike the WebM or Theora implementations). A 720p video at 24FPS seems to work fine, but a 1440p 60FPS one will stutter a lot.
• Data is read from memory instead of file.

[fire]

Did you investigate EIRTeam/EIRTeam.FFmpeg@49efa2a?

[DeeJayLSP]

Did you investigate EIRTeam/EIRTeam.FFmpeg@49efa2a?

I was focusing on getting things working at an acceptable way first. Using a shader would complicate things more (I have no idea of how to implement it).

Edit: I figured out how to use yuv2rgb, but the performance gain was minimal. Despite CPU usage showing to be slightly lower than Theora.

I am sure it is related to conversion because telling the video to decode without converting anything doesn't drop performance at all.

Edit 2: Seems like generating the Image has a bit of fault too. I don't understand the problem since it was mostly copypasted from VideoStremTheora.

Edit 3: Moving Image generation to update() significantly improved performance, although there is a noticeable stutter at the beginning that affects video.

Edit 4: Moving yuv2rgb out of the video callback solved the stutters. With performance issues resolved, only the file buffer issue is left.

[berarma]

I think there's value in having a well supported and highly compatible codec in the core. I've created godotengine/godot#101958 to address the issues with Theora video streams.

[DeeJayLSP]

I think there's value in having a well supported and highly compatible codec in the core. I've created godotengine/godot#101958 to address the issues with Theora video streams.

Guess I was wrong about libtheora being broken. It was the majority of implementations, including Godot's.

Oh well, I had a good time making that pl_mpeg implementation. Maybe I'll get it working at the best possible state and release as a custom module.