Firefox Can Only Play a Still Frame of Video

[ElliotKillick]

Firefox only seems to be able to play a single frame of video. I've already diagnosed why this is happening with both GStreamer and FFmpeg test commands leading to the same root issue.

Firefox doesn't seem to like the YUV video format but when I tried an MJPEG stream (stream of JPEG images, which is a common format for webcams) it works fine and plays the video back seamlessly.

We could add a conversion to the MJPEG format but I tested that and it results in big latency increase (not to mention the added attack surface). Additionally, to do that GStreamer actually relies on a libav (fork of FFmpeg) conversion algorithm that got bindings added to it for GStreamer. This is the element in question: https://gstreamer.freedesktop.org/documentation/libav/avenc_mjpeg.html. This means it's not packaged in Fedora due to the patent issues with FFmpeg.

So, with the problem diagnosed I'm going to try and make quick work on finding where the bug is so it can be patched. Firefox is the only application I've tested that has had problems so it may be a Firefox bug with YUV formats. I looked it up and Firefox should have full support for YUV webcam video just like Chromium but that doesn't appear to be the case.

Going to try and fix this ASAP because I hate to leave Firefox at a disadvantage.

[ElliotKillick]

I tested on another computer configuring my webcam to use the YUV format my webcam supports (with v4l2-ctl) and then seeing if Firefox would play the video and it worked perfectly fine which indicates this isn't an issue with Firefox.

As was stated in the original issue, I have both FFmpeg and GStreamer commands can reproduce this issue so they are definitely not the root cause.

Both of these play raw YUV test video and put it into the v4l2loopback driver and they both fail to make the video play in Firefox:

gst-launch-1.0 videotestsrc ! tee ! v4l2sink device=/dev/video0
ffmpeg -f lavfi -i testsrc -f v4l2 /dev/video0

Conversely, here is an example of a working pipeline that Firefox will play the video of (converts YUV into MJPEG):
gst-launch-1.0 videotestsrc ! avenc_mjpeg ! v4l2sink device=/dev/video0

All of these were tested in Firefox with the webcam.html file included in this repo to make sure there are no discrepancies there either.

This must mean it is indeed a problem with v4l2loopback. I have a sneaking suspicion that this problem may be related to this issue: umlaeute/v4l2loopback#190. The [video4linux2,v4l2 @ 0x5581164f6280] Dequeued v4l2 buffer contains 3112960 bytes, but 3110400 were expected. Flags: 0x00000001. warnings mentioned in this issue were also constantly popping up for me when I was using FFmpeg to preview the video being streamed into v4l2loopback device for testing. So my theory is that perhaps Firefox doesn't know how to handle this small oddity with how v4l2loopback is spitting out the video stream thereby causing it to not play past the first frame in Firefox.

In any case, I think it's definitely been narrowed down to an issue with v4l2loopback at this point.

Just want to clarify this issue's title as well ("Firefox Can Only Play a Still Frame of Video"): Upon giving Firefox access to the webcam, it will only display the first frame of video and then it will just be stuck displaying that first frame and won't display any more frames. Basically, it will just be a still image. If the page is reloaded, then it will take a new single frame of the webcam video feed and be stuck on that.

[DemiMarie]

Does Firefox change the format to MJPEG, or does it handle a source that only supports YUV?

[ElliotKillick]

Firefox supports both MJPEG and YUV formats. I'm not sure if when Firefox gets YUV it (behind the scenes) converts the YUV to MJPEG in order to display it or what happens there.

I know Chromium uses libyuv for handling YUV video. Although, through reading the project description in that link it actually goes the other way around from MJPEG to YUV:
"Convert to YUV from webcam formats for compression"

[ElliotKillick]

Note in relation the v4l2loopback issue I referenced earlier: It seems like for that person FFmpeg stops outputting video after that warning but when I was testing playing back video being delivered to the v4l2loopback that warning just kept popping up constantly with every frame that played, however, the video preview kept playing.

Just to clarify, I was of course using the GStreamer pipelines in this project to stream the video over to the video receiving VM into v4l2loopback where it would be made available by reading the /dev/video0. Using FFmpeg (ffplay), I then read from that /dev/video0 device and displayed the video in a preview window. It was at this point, those aforementioned warnings started constantly popping up while the video was playing.

[ElliotKillick]

@DemiMarie submitted the PR to fix this in v4l2loopback as outilined in the issue: umlaeute/v4l2loopback#435

Just waiting for it to be merged now.