VideoStreamTheora: Identical frames make the audio buggy

[DeeJayLSP]

Godot version

Both 3.5 and 4.0 beta 1

System information

Kubuntu 22.04

Issue description

On videos that start with a black screen, the audio plays incorrectly.

This bug is much more noticeable if the keyframe interval is set to a high value (through -g or -g:v) such as 300 (10 times the frame rate, the max recommended by Theora guides), where the audio only begins playing after the first non-black frame appears, then randomly starts trying to seek to the correct position. This won't happen in video players such as VLC or mpv.

On videos that don't start with a black screen, or have a keyframe interval set to 1 (which will result in a very heavy file) the audio begins playing normally.

(Seriously, increasing the keyframe interval is simply an amazing way to reduce Theora file sizes, at the cost of having some very minor motion artifacts, but you can get a video of quality 8 at the same size as quality 6 at low keyframe interval)

Steps to reproduce

1. Download the Bink file at this video's description (this is the exact same video I used to find this bug);
2. Convert it to Ogg Theora using FFmpeg (-g:v set to a high number such as 300 is very important)
3. Drag the video into Godot and play it.

• Also try using a low keyframe interval, it will noticeably reduce the delay.
• FFmpeg uses a default value of 12 for the keyframe interval.

Minimal reproduction project

I'm afraid of inserting a branded video here.

[DeeJayLSP]

Probably related to #21568, but in this case the video begins with identical frames.

[0000mz]

Hi @DeeJayLSP . Could you paste the ffprobe log of the video that's causing the problem?

[DeeJayLSP]

Input #0, ogg, from '8_1K300ff.ogv':
  Duration: 00:00:22.76, start: 0.000000, bitrate: 1114 kb/s
  Stream #0:0: Video: theora, yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], 29.97 tbr, 29.97 tbn, 29.97 tbc
    Metadata:
      encoder         : Lavc58.134.100 libtheora
  Stream #0:1: Audio: vorbis, 48000 Hz, stereo, fltp, 160 kb/s
    Metadata:
      encoder         : Lavc58.134.100 libvorbis

I really don't see a problem there.

[DeeJayLSP]

There are 2 encoders that I use: FFmpeg and ffmpeg2theora.

For the first, FFmpeg does both the decoding and encoding.

For the second, FFmpeg does the decoding, but encoding is done by your local libtheora and libvorbis. While it has some (really avoidable) bugs due to expecting an old version of FFmpeg, as ffmpeg2theora hasn't been updated since 2016 (you also have to do some things like converting the video to another encoder like H.264 or FFV1 before that), its --optimize argument is really worth it. It manages to output a video with 5-7% increased size, but the significantly increased quality makes it worth it.

Despite that, a video encoded with both yield the same problem.

[Calinou]

(Seriously, increasing the keyframe interval is simply an amazing way to reduce Theora file sizes, at the cost of having some very minor motion artifacts, but you can get a video of quality 8 at the same size as quality 6 at low keyframe interval)

This will be interesting to mention in the Playing videos documentation if we can manage to fix this bug 🙂

[DeeJayLSP]

Well, as long as a video doesn't start with multiple identical frames there will be no bug.

[DeeJayLSP]

The bug likely happens on this while loop (sorry for using 3.x, I wanted to compare to WebM playback on the same build)

Tried messing around (took away && !audio_done), the audio actually played before the first non identical frame showed, but it was buggier than before and didn't play along the video.

I believe the bug may be caused by whatever determines the value of !audio_done.

See the comment below this one.

There's a comment right below the printf() function mentioned below, that... mentions keyframes.

[DeeJayLSP]

I found a printf() lying at line 483 (outside of the loop I mentioned), decided to uncomment it. The offset is interesting:

It's like the video is frozen until the first different frame appears. The sound also only begins playing at that time.

WebM (both VP8 and VP9) is free from this problem.

[DeeJayLSP]

I don't have a deep understanding of C++, but there is a behavior that I understood well:

• Audio doesn't begin playing on the first frame, only on the second frame;

• By second frame, I meant the second time the printf() at line 483 is reached;

• If the video starts with identical frames, that second frame is only reached when the video jumps out of the identical frames, or when one of those identical frames is a keyframe;

• This means that in a 30fps video with a keyframe interval of 300 frames, given the identical frame lasts over 10 seconds, the second frame will come out at precisely 10 seconds.

[hungrymonkey]

@DeeJayLSP

Ogg-Theora is an awkward file format to work with. The problem is that the format encourages a lot of moving parts. I believe I creator made those moving parts to encourage easier encoder design, but this stuff is hard.

https://web.archive.org/web/20140502001132/https://people.xiph.org/~xiphmont/lj-pseudocut/o-response-1.html

https://lwn.net/Articles/382478/

You can read about my attempt I didn't bother to upstream awhile ago.

https://hungrymonkey.github.io/seeking-three-moving-windows-ogg-theora/

[FlankeR-46]

Hello. I use version 4.0.stable and I have a similar problem. The sound is noticeably lagging behind the video. Is there any information whether this issue will be resolved? Thanks.

[Ktar5]

I encountered the bug, I used the -g:v 1 flag when converting my video to ogv in ffmpeg and everything worked. Posting in case anyone comes across this issue and needs a solution before its fixed.

[Calinou]

I encountered the bug, I used the -g:v 1 flag when converting my video to ogv in ffmpeg and everything worked. Posting in case anyone comes across this issue and needs a solution before its fixed.

Note that -g:v 1 essentially disables inter-frame compression (since every frame is a keyframe), so file size will be much higher. I'd try using higher values until you encounter buggy audio.

[berarma]

For the second, FFmpeg does the decoding, but encoding is done by your local libtheora and libvorbis. While it has some (really avoidable) bugs due to expecting an old version of FFmpeg, as ffmpeg2theora hasn't been updated since 2016 (you also have to do some things like converting the video to another encoder like H.264 or FFV1 before that), its --optimize argument is really worth it. It manages to output a video with 5-7% increased size, but the significantly increased quality makes it worth it.

I've come across this in the Theora source code and I thought I'd share the info here.

The Theora documentation says the default encoding speed level is 0 (the slowest, more quality), but the library is using speed level 1 as default. ffmpeg2theora is setting speed level to 0 when --optimize is used.

I'll try to get this change into ffmpeg, either as a switch or always speed level 0.