Low performance when receiving high quality remote video on HL2

[eirikhollis]

When receiving a 720p30 remote stream on HL2, the application framerate drops severely. If the remote peer connects and does not publish video, the framerate stays at 60.

I've also tested with a 320x240 15fps video stream, and while the framerate does drop to ~55, it is not nearly as bad as high quality streams.

Is this due to missing H264 hardware decoding?

[djee-ms]

Is that related to #151 by any chance? Could you please try to cherry-pick the workaround from the last message of @fibann, to confirm?

[eirikhollis]

So tried commenting out the two lines in @fibann's workaround, namely

//LocalVideoTrackInterop.LocalVideoTrack_RegisterArgb32FrameCallback(
//    _nativeHandle, _interopCallbackArgs.Argb32FrameCallback, _selfHandle);

//PeerConnectionInterop.PeerConnection_RegisterArgb32RemoteVideoFrameCallback(
//    _nativePeerhandle, _peerCallbackArgs.Argb32RemoteVideoFrameCallback, self);

and I did indeed see a difference. Instead of framerates around 10-30, they were more consistent around 40-60. There were some dips down to as low as 5-10, but it recovered to 60 in a few seconds.
I'll continue testing with multiple incoming video feeds and see how it affects the performance.

[eirikhollis]

Performance is definitely lower on HL2 than HL1 when there are multiple streams in, using the same configuration for both.
HL2 seems better than HL1 in the start, but as more connections join on the call, the degradation is a lot quicker on HL2 than HL1.

[djee-ms]

Interesting feedback. I assume you are still using a peer connection per user, so each "more connections join on the call" is actually a new PeerConnection object being created, correct? This doesn't excuse the bad performance, but it's obviously not the best approach, and hopefully supporting multiple audio/video tracks (#152) with a media server and a single PeerConnection per device should I think yield better performance, as it doesn't duplicate all the PeerConnection machinery, only the necessary audio/video tracks.

[eirikhollis]

Yes, I still use a new PeerConnection object per individual stream that is received. If there is anything you'd like me to test with that setup, I'll be happy to help.
Looking forward to test the multiple audo/video track feature when that is ready!

[fibann]

Not much to do here - #151 is being worked on, and hardware decoding is tracked as #27, so closing this.