Remote audio starts clipping on HoloLens when simultaneously receiving large video streams

[eirikhollis]

Describe the bug
When establishing peer connections with a large amount of remote peers and receiving video from all of them, the audio on the HoloLens 2 starts clipping. It's as if it skips a couple hundred milliseconds every other second or so. This is dependent on the WebRTC load on the device. I haven't experienced this in a one-to-one connection yet, but I did experience it while connected to two remote peers, where one of them sent a 1080p30fps stream and the other a 240p15fps stream. This is also present on the HoloLens 1, but the tolerance for the amount of WebRTC load present before the audio starts clipping is a lot lower, probably due to a weaker CPU.
I'm therefore assuming that a thread on the device is saturated due to a high workload on video, which then in return affects audio quality.

To Reproduce
I'm guessing it could be reproduceable in a one-to-one connection if the video received is of a significant enough size, maybe 1440p30fps or higher?
Reproduceable with multiple peers where most, if not all, send video as well as audio.

Expected behavior
Clear audio. The audio sent from the HoloLens 1/2 is clear and without issue. The audio received on all other peers in call is also clear and without issue.

Environment
Issue on HoloLens 1 and 2.
Remote peers are on a variety of devices; React web-app in browsers Edge/Chrome/Safari, mobile devices with Edge/Chrome/Safari, and also from Unity.
Unity Version: Present in all 2019.3.x versions (and probably earlier)

Additional Information
I would be willing to send an audio clip on slack if you haven't heard the issue yet.

[kspark-scott]

Hi @eirikhollis , these symptoms sound very similar to symptoms we saw a while back. We described it as a "stutter", so perhaps that's not quite the same as clipping. In our case it turned out to be the garbage collector running major collections every second or two, stop-the-world collections taking 70-80 ms each. It only happened under load because only then did we fill the heap fast enough.

The problem was not a leak, but rather unnecessary allocations -- allocating a byte buffer for each frame rather than re-using a pre-allocated one.

Your case may be different, but worth checking perhaps.

[eirikhollis]

@kspark-scott Thanks for the input.

I tried looking into garbage collection and decided to completely turn of displaying the remote videos in the UI of the HoloLens. All video-displaying related code got disabled and I commented out any line of code that listened to the Track events within the DLL.

I tried first with the Video Transceiver on each Peer Connection set to Transceiver.Direction.Inactive (thus not negotiating video), and the clipping (or stuttering as you say, probably a better word) was almost completely gone. We could hear it a bit with 7+ audio connections, but only two times total, and very minor. Unlike the every (other) second or so like before.

When setting the Video Transceiver direction to Transceiver.Direction.ReceiveOnly however, the stuttering returned, just as much as it used to be before. This is without actually displaying the remote video streams, just receiving them within the DLL and ignoring the TrackAdded events.

I've had the Incremental Garbage Collection feature within Unity Player Settings enabled during testing, and I'll try disabling it tomorrow, might have an effect.

Any other tips or tricks?

[kspark-scott]

No, I'm afraid I have no other ideas for what might cause this particular issue. The garbage collector impact wasn't immediately obvious so we spent some time hunting for plausible potential causes and never came up with anything more specific than general suspicions of some kind of thread saturation or contention. It was the very regular periodicity of the audio interruptions that implicated the garbage collector -- it's not a behavior that is a likely outcome of a pegged CPU core.

[fibann]

@kspark-scott from your previous post:

The problem was not a leak, but rather unnecessary allocations -- allocating a byte buffer for each frame rather than re-using a pre-allocated one.

Were the unnecessary allocations in your app or MR-WebRTC? We have already tracked work to reduce the CPU load when decoding video (#151, #27) but if there are other opportunities to improve performance it would be great to know.

[kspark-scott]

@fibann, you were actually the one that fixed the unnecessary allocation that was causing this :-)

12bf954

We are using the VideoBridge class from the sample app with virtually no customizations because it does the job we need and it's not immediately obvious to me how to improve on it.

We are not currently aware of other performance related issues that you are not already tracking.

[fibann]

Thanks @kspark-scott, I remember that issue, just double-checking that there weren't others that we weren't aware of.

I am going to close this since there is not much else we can do, it seems plain performance degradation due to saturated CPU. See linked issues above for ongoing work on perf.