VS Code Video

Notice

This repository started off as an attempt to build a VS Code extension to play video files in VS Code using the web browser API. At the time I attempted this, VS Code did not ship with the necessary codecs.

At some point, that looked like it has changed:

https://code.visualstudio.com/updates/v1_71#_ffmpeg-codecs-support

These release notes show that VS Code started shipping with extra FFmpeg codecs.

This got me wondering whether it was now possible to build the extension after all. But before I got into researching that, VS Code has started shipping with the video preview feature itself!

https://twitter.com/mattbierner/status/1570857070156746754

That's the best possible outcome and it makes this whole endeavour obsolete, so this document is left as a historical artifact and the repository is archived.

Installation

Install from source because the extension uses proposed APIs and so it cannot be published to the VS Code extension marketplace.

Prerequisites

The user has to have Google Chrome installed for this extension to work.

Limitations

VS Code uses Chromium which does not bundle media codecs with it, thus the webview cannot be used to play the media directly.

microsoft/vscode#54097

This extension uses Puppeteer Core and local installation of Chrome (if any) to play the video in Chrome and controls the headless Chrome instance so that the video frames are transferred over to the webview.

Running

Use VS Code Insiders! Press F5 to start debugging the extension.

Resources

https://code.visualstudio.com/api/extension-guides/custom-editors

https://github.com/microsoft/vscode-extension-samples/tree/master/custom-editor-sample

To-Do

Try using the screencast API for a better performance (but what about quality?)

Add playback controls which use vscode.postMessage to the editor provider

Turn the progress element into a scrubbar which also uses postMessage

Load the JS and the CSS from bundled extension resources like example shows

https://github.com/microsoft/vscode-extension-samples/tree/master/custom-editor-sample

Figure out how to make the video have its native dimensions in Puppeteer

Perhaps I need to open a data:text/html page with the video or maybe event a temporary local file (in case the data URL cannot play the video) so that user agent styles are not forcing the video to fit the viewport or whatever they are doing right now.

Handle the user not having Chrome installed using an error message

See if it would be possible to use the Uint8Array for better perf

Maybe I could use a canvas instead of the image element and blit the image data to it.

See if a custom MediaSource could be devised for video element to be used

This would simplify the HTML and we could use native controls but probably would be very complex if not impossible due to having to handle the container.

Fall back to Microsoft Edge in case Chrome and Chromium are not installed

Use Playwright or see if Puppeteer Core still works as it is all CDP in the end.

Consider adding audio support (mainly in the UI, Chrome plays the audio)

Publish the extension once the proposed custom editor API hits stable

Install Chrome or Chromium in the GitHub Actions workflow to make it go