Twilio Video Processors is a collection of video processing tools which can be used with Twilio Video JavaScript SDK to apply transformations and filters to a VideoTrack.
- Virtual Background
- Background Blur
- Twilio Video JavaScript SDK (v2.13+)
- Node.js (v14+)
- NPM (v6+, comes installed with newer Node versions)
Install from a local directory.
# package-folder contains the package.json
npm install /local-path-to-package/package-folder
Using this method, you can import video-processors like so:
import * as VideoProcessors from '@twilio/video-processors';You can also copy video-processors.js from the dist folder and include it directly in your web app using a <script> tag.
<script src="https://my-server-path/video-processors.js"></script>Using this method, video-processors.js will set a browser global:
const VideoProcessors = Twilio.VideoProcessors;In order to achieve the best performance, the VideoProcessors use WebAssembly to run TensorFlow Lite for person segmentation. You need to serve the tflite model and binaries so the SDK can load them properly. These files can be downloaded from the dist/build folder. Check the API docs for details and the examples folder for reference.
Please check out the following pages for example usage. These processors are only supported on the Chrome browser at this moment and will not work on other browsers. For best performance and accuracy, we recommend that, when calling Video.createLocalVideoTrack, the video capture constraints be set to 24 fps frame rate with 640x480 capture dimensions. Higher resolutions can still be used for increased accuracy, but may degrade performance, resulting in a lower output frame rate.
- Video Processor execution will result in a significant increase in CPU usage.
- Precision on segmentation mask can be poor on certain conditions such as uneven lighting and increased body movements.