Method to directly render the VideoDecoder to a Canvas element

[koush]

Currently, sending data from the VideoDecoder to Canvas involves:

• VideoDecoder.decode
• VideoFrame.createImageBitmap
• createImageBitmap again to flipY the imageOrientation (it seems that Chrome currently doesn't support the ImageBitmapOptions parameter on VideoFrame)
• CanvasContext.transferImageFromBitmap

Under the hood, this I suspect this converts VIdeoFrame YUV to ImageBitmap RGB, rather than directly rendering the YUV planes of the video frame in the webgl canvas. (Please let me know if I'm wrong)
There's also some lifecycle management on the frames and bitmaps to close and destroy them.

Ideally, the video frames (or perhaps decoder) should have a method to directly render to a canvas. This is similar to how Android's decoder operates, and for lack of better description "feels" like it is closer to achieving zero copy.

[sandersdan]

Edit: ImageBitmapOptions is in the spec and in Chrome's implementation. If it doesn't work in Chrome Canary please file a Chromium bug.

In Chrome's implementation, createImageBitmap does indeed do a conversion to RGB. There are optimizations we still plan to make, but it's not yet certain what faction of cases will continue to require a conversion.

There isn't currently a way to bind WebGL textures to images, but this is something that operating systems can usually do. I've heard that WebXR is investigating whether there is a path forward there.

I'm hopeful that WebGPU will be more amenable to this sort of usage. There has been some discussion but no agreed plan, see gpuweb/gpuweb#625.

[koush]

Thanks for the implementation details, that is good to know. I look forward to any future optimizations.

Looking into VideoFrame interface, it does seem like the YUV Planes are accessible for reading (though it presumably requires a memcpy). I can use that to render the YUV onto a webgl canvas context. This will likely be more efficient than performing a userspace YUV->RGB->flipY->render.

https://wicg.github.io/web-codecs/#plane

If there's a better method to render h264 nal units directly to a canvas than what I described above, please let me know!

[koush]

I made the changes to draw the VideoFrame yuv planes via webgl and can confirm that the rendering is significantly faster. Latency and CPU usage both decreased dramatically.

Setting up and rendering via a GL context is more involved, so convenience methods to do this would be great. Right now, the createImageBitmap/transferImageFromBitmap is deceptively expensive and slow. Problematically, it is seemingly the fastest/direct way to draw to the screen/canvas.

[koush]

Update:
I ran into a few issues with decoding and then rendering to a canvas.

On ChromeOS, and other hardware accelerated platforms, VideoFrame.planes may be null. Furthermore, VIewFrame.createImageBitmap is returning blank/black frames.

This means on some (indeterminable at runtime) platforms, the decoder is completely non-functional, as the frames can't be retrieved in any fashion.

Chrome 86.

[jchen10]

For ChromeOS issue, there is a bug at https://bugs.chromium.org/p/chromium/issues/detail?id=1137947
Could you upload your error log in chrome://gpu there? thanks!

[chcunningham]

+1 to jchen's suggestion.

For folks reading along: the case of null planes for hw-decoded frames is a known limitation of chrome's current impl, but the bug in createImageBitmap() breaks the intended workaround. In addition to getting the bug fixed, we're planning to spec an API for format conversion (e.g. null -> i420).

[koush]

@chcunningham Unfortunately, even the createImageBitmap is an unusable workaround for my use case, as it does an unaccelerated YUV to RGB conversion and then memcpy that onto the canvas. There is significant latency and freezing on low power devices. It's actually faster to use a WebAssembly h264 decoder, because the YUV planes are accessible and can be rendered directly with WebGL.

The ideal solution as mentioned in title is just a way to render directly to a canvas from a decoder. The conversions, while nice to have, are a roundabout way to get the decoder rendering to an element, which I imagine will be the primary use case.

[koush]

will get a chrome://gpu for the bug shortly.

[chcunningham]

The ideal solution as mentioned in title is just a way to render directly to a canvas from a decoder. The conversions, while nice to have, are a roundabout way to get the decoder rendering to an element, which I imagine will be the primary use case.

Agreed. We're tracking these optimizations here
https://bugs.chromium.org/p/chromium/issues/detail?id=1141182

[AlexVestin]

Agreed. We're tracking these optimizations here

Can this also apply to have a zero-copy from a canvas to a VideoEncoder, in cases where hardware acceleration is available?

[chcunningham]

@koush and others, we have some updates!

• we've just landed support for painting VideoFrames via drawImage and texImage2d. This will release in canary channel shortly (check here: https://chromiumdash.appspot.com/commit/9229440513bd0995595a6e3ab72686d3f7f1c8d6)
• we've also done some work to expose extra pixel formats (e.g. NV12) and map gpu backed memory for copy access. I expect you really just want drawImage^, but just a FYI that your chromeos decodes may now give you pixel data depending on your stream's underlying pixel format

Please let us know how it works!

Can this also apply to have a zero-copy from a canvas to a VideoEncoder, in cases where hardware acceleration is available?

@AlexVestin, most encoders will require that we at least do an RGB -> YUV conversion before encoding. This conversion can happen on the GPU, so its not awful. Still, there are some additional copies in the path of Canvas -> ImageBitmap -> Encoder which could be better optimized. Can you share more on your use case? I may be able to suggest a more efficient path.

[koush]

• we've just landed support for painting VideoFrames via drawImage and texImage2d. This will release in canary channel shortly (check here: https://chromiumdash.appspot.com/commit/9229440513bd0995595a6e3ab72686d3f7f1c8d6)

It looks like you can feed VideoFrame directly to texImage2d! This is great!
https://chromium.googlesource.com/chromium/src.git/+/9229440513bd0995595a6e3ab72686d3f7f1c8d6%5E%21/#F9

let frame = new VideoFrame('ABGR', [argbPlane], vfInit);
gl.texSubImage2D(gl.TEXTURE_2D, 0, 0, 0, gl.RGBA, gl.UNSIGNED_BYTE, frame);

[BenV]

@chcunningham I implemented this in a test app this morning, and unfortunately I am seeing quite a bit of overhead when using VideoFrame as opposed to an HTMLVideoElement with texImage2d to process frames from my webcam on both PC and Mac. It takes roughly twice as long when using the VideoFrame method. Is this expected?

FWIW I've done quite a bit of benchmarking on this stuff while trying to find the most optimal way to manipulate a camera feed, and these numbers look pretty close to what I was seeing for the total time when using window.createImageBitmap on the HTMLVideoElement and then passing the resulting ImageBitmap into texImage2d.

[koush]

Have implemented the new VideoFrame canvas rendering and can confirm that this is working with hardware acceleration on my Mac now. Prior to this Canary build, the previous method had no way to render the frame, as createImageBitmap was returning a blank bitmap.

[AlexVestin]

@chcunningham
The createImageBitmap call with a canvas averaged at around 10ms, so I assumed it was copying the frame back from the GPU, but I might have been mistaken about that.

Can you share more on your use case? I may be able to suggest a more efficient path.

The use case is encoding frame by frame from a canvas for animation rendering, in cases where the user is not be able to encode it in real time

[dalecurtis]

We're merging the video element and VideoFrame paths, so hopefully you won't see any differences shortly. There are a couple bugs with the VideoFrame path on Canary that I hope to resolve this week. There may be some slowness introduced on macOS where previously we weren't waiting on the appropriate sync tokens, but it shouldn't be any slower than the video element path.

[dalecurtis]

Also, @BenV if you have a public page for benchmarking I'd love to use it for experimenting with changes to this path.

[BenV]

@dalecurtis I don't have a public page currently but will work on getting one set up and will keep you posted. Thanks!

[BenV]

@dalecurtis While I still plan on creating a public benchmark I'm happy to report that with the latest Chrome Canary I no longer see any speed penalty in my internal test, and the VideoFrame path is actually a bit faster for me on both Mac/Windows.

I do seem to be losing my WebGL context fairly frequently on my Windows machine, although this could be related to some other issue with the Canary build. My entire window (including the Chrome UI) flashes to black and I get WebGL: CONTEXT_LOST_WEBGL: loseContext: context lost in the console. Right above this in the chrome_debug.log I see: [11500:35844:0225/153241.044:ERROR:gpu_init.cc(426)] Passthrough is not supported, GL is disabled, at that point I am unable to restore the context or create any more WebGL contexts in that tab. This happens pretty consistently for me after running my application for about a minute, and does not seem to happen with the previous HTMLVideoElement texImage2d path. I wonder if it has to do with the timing of calling VideoFrame.close. I will see if I can narrow this down a bit.

Sorry if this is the wrong place to ask, but are there plans for making VideoFrame objects transferable? It would be ideal to be able to send them to a worker for rendering. Thanks so much for your work on this, we're very excited about these new APIs.

[dalecurtis]

@BenV Great to hear! Our perf tests indicates the new video-element/VideoFrame path is 12-60% faster, so good to hear that's matching your results.

Regarding the black flashes, do you have any crashes in chrome://crashes when this happens? Usually it means the gpu process has crashed. If you provide the crash ids I can look into it.

VideoFrames should already be transferrable. Are you seeing issues doing that? @tguilbert-google @sandersdan

[BenV]

@dalecurtis I am not seeing any problems with transferring VideoFrames, just did not realize that was already possible, that is great news!

I do not see anything in chrome://crashes, I did verify that this is also happening in the nightly build. I'll keep trying to narrow this down and keep you posted. It does appear to work fine for my 2D canvas code path on this machine -- the issue only occurs with WebGL/texImage2D. Is there anything else you would like me to do that would be helpful?

Things I have already quickly tried without success are:

• Using gl.flush/gl.finish before closing the VideoFrame
• Delaying the call to VideoFrame.close
• Upgrading from VideoTrackReader to MediaStreamTrackProcessor

Edit: Just noticed my GPU process memory footprint is growing indefinitely while using VideoFrame with texImage2D, so that's a clue. Quite possibly a problem on my end, but I believe the only difference is passing the VideoFrame instead of the HTMLVideoElement. Will keep investigating.

[dalecurtis]

Ah sounds like OOM then, could be a leak on our side. You should be able to call texImage2D(frame) and then immediately call frame.close() -- if you have a test page that reproduces let me know and I'll take a look.

[tguilbert-google]

(Oops, misclicked closing the issue)

A quick clarification: transferrable VideoFrames have not yet landed, but the CL is out for review.

[dalecurtis]

I think I can reproduce the leak with my simple test page, https://storage.googleapis.com/dalecurtis/webgl-camera.html will take a look.

[BenV]

Was just about to link this repro case but it happens for me on yours as well: https://codesandbox.io/s/videoframe-webgl-test-vjtb3

Thanks again for being so responsive on this issue, it is very much appreciated! Please let me know if I can do anything else to test or help out in any way.

[dalecurtis]

https://bugs.chromium.org/p/chromium/issues/detail?id=1182476 tracks the leak. Still digging.

[BenV]

I can confirm the leak is fixed for me in the Chromium nightly, nice work, and thanks again!

[chcunningham]

I've now updated the explainer to recommend drawImage and texImage for painting VideoFrames. Also, I've filed #158 to update the html spec to include VideoFrame as a CanvasImageSource.

I think all issues in this thread are now addressed, but please re-open if I've missed anything. Thanks so much for the feedback!

[rcunning]

Ah sounds like OOM then, could be a leak on our side. You should be able to call texImage2D(frame) and then immediately call frame.close() -- if you have a test page that reproduces let me know and I'll take a look.

@dalecurtis I am fighting a bug where calling frame.close() immediately after texImage2D produces a black texture, and only on Windows (Chrome 113) machines in our testing. We are processing 64 frame chunks in a web worker, and typically on the problematic machines the last 2-5 frames in the sequence end up black. Adding a 10 ms timeout before calling frame.close() resolves the issue, but this is not desired since it can slow down the decoder and timers add overhead. Do you have any ideas for ways to prevent this issue?

[dalecurtis]

Please file an issue at https://crbug.com/new with the chrome://gpu details and simple reproduction if it doesn't reproduce using https://w3c.github.io/webcodecs/samples/video-decode-display/

[rcunning]

Please file an issue at https://crbug.com/new with the chrome://gpu details and simple reproduction if it doesn't reproduce using https://w3c.github.io/webcodecs/samples/video-decode-display/

I just wanted to follow up here so you know the outcome @dalecurtis -- the issue was there was a race condition between closing the decoder and calling textImg2D with a few of the frames at the end of the segment of video being decoded, resulting in occasional cases of closing the decoder before textImg2D on the frames. I would think we would get some sort of error message in this case (would have saved a lot of time debugging!), but instead were ending up with empty textures.

[sandersdan]

That's a bug; frames should remain valid even if the decoder is closed. (On some platforms, implementations may need to keep the decoder alive as long as there are outstanding frames.)