Receiving multiple remote tracks/streams

[drejx]

Hi,

I've been integrating the MixedReality-WebRTC SDK and as I've been going along I've hit a stumbling block I hope someone can help with, namely how are you able to receive multiple remote tracks? All the track related logic appears to only ever consider 1 remote track. For example neither of these events contain a track/streamID:

OnTrackAdded(TrackKind trackKind) 
OnI420ARemoteVideoFrameReady(I420AVideoFrame frame)

Are multiple remote tracks on the same peer connection possible with the SDK? Or is there an expectation to use a new peer connection for each desired remote track? I imagine that would use way more resources that a single peer connection with multiple remote tracks.

To give an idea of my use case:

1. I make a peer connection to a relay server using a unique connection ID.
2. If any other (remote) peers connect to the relay server with the same connection ID then the relay server notifies me that a new remote track ID is available.
3. On my peer connection I go through the offer/answer process (the updated SDP includes the newly added remote track).
4. I receive a "TrackAdded" event with the new remote track/stream included.

Disclaimer: I started a few weeks ago with the webrtc-uwp-sdk SDK and have the above scenario working with that SDK because it supports multiple remote tracks. It includes an OnTrack event with the following signature:

public delegate void RTCPeerConnection_OnTrackDelegate(IRTCTrackEvent Event);

where IRTCTrackEvent contains:

public interface IRTCTrackEvent
{
    IRTCRtpReceiver Receiver { get; }
    IMediaStreamTrack Track { get; }   ====> this has track ID and stream source data
    IRTCRtpTransceiver Transceiver { get; }
}

I'm surprised I don't see the same setup in the MixedReality-WebRTC SDK and would appreciate any insight one can send my way. So far I find the MixedReality-WebRTC is much easier to use than webrtc-uwp-sdk and a big bonus is it offers more than just a UWP platform (e.g. Unity integration & Desktop apps).

Thanks!
Andrej

[djee-ms]

Yes this is by design due to lack of resources, for v1.0 we did the simple thing of supporting a single remote track per peer connection. For v2.0 we will support multi-tracks for all media (audio and video, local and remote). Currently on master this has been committed for local video tracks only though, and the 3 other ones still support a single track per peer connection only. You can either wait for support in v2.0 (or on master as soon as it comes, probably several weeks though with the holidays etc.) or go with multiple peer connections for now as a workaround.

[drejx]

Aah okay. It's good to know the timeline when multi-tracks will be available for remotes. In the meantime I will try using separate peer connections for each new remote connected after the first one.

One thing which could be a problem for the additional peer connections is the SDP would need to be set to receive only, i.e. I only want to receive video from the remote, I don't want to duplicate sending my local video out multiple times. I don't see any way in the current API to do that...I can access the raw local SDP string when the LocalSdpReadytoSend event is called, and modify the raw SDP. But it looks like the SetLocalDescription call is done internally and thus won't use my modified raw SDP string.

Just a sidenote that on the project's github readme.md one of the bullets is:

Enables multi-track real-time audio / video / data communication with a remote peer

Which led me to believe that remote multi-tracks is supported. Friendly suggestion to maybe add a note that it will be available in v2.0 :)

p.s. As this SDK gets more traction, which I really think it will, there will definitely be more engagement from the community so I hope you get a whole bunch more support/resources ;)

[drejx]

Thinking about the above workaround, if I create a new peer connection for each additional remote track and I don't call Peer.AddLocalVideoTrackAsync() I would imagine that the SDP would be set to recv only. I haven't tried this yet, but will see.

[djee-ms]

Yes that should work.

[drejx]

Hi Jerome,

Just leaving a quick update that I've implemented the workaround, i.e. for every remote peer, except for the 1st one, I create a new peer connection instance to receive the remote stream.

The key was to not call Peer.AddLocalVideoTrackAsync() on those remote peer connections and call Peer.SetLocalAudioTrackEnabled(false) so that not audio/video was being sent out. I also had to mute any incoming audio on those additional remote tracks of course :)

Cheers,
Andrej

[djee-ms]

Hi @drejx, glad to hear that worked, and thanks for the feedback on how you did it, I am sure this can help other users!

Hopefully we can land proper multi-track support soon, although that proved more difficult than expected and I am still working through technical issues and understanding the details of how Unified Plan works. I'll post an update when it's ready.

[sesd777]

Hi Jerome,

Just leaving a quick update that I've implemented the workaround, i.e. for every remote peer, except for the 1st one, I create a new peer connection instance to receive the remote stream.

The key was to not call Peer.AddLocalVideoTrackAsync() on those remote peer connections and call Peer.SetLocalAudioTrackEnabled(false) so that not audio/video was being sent out. I also had to mute any incoming audio on those additional remote tracks of course :)

Cheers,
Andrej

Hi @drejx,

I am basically trying to achieve the same as you - a multipeer conference.

My current structure looks like that:

• a new (native) PeerConnection is beeing created for each user of a session, i.e. a session of 2 people (user A and user B). Because of that I intent to create the offer for the local PeerConnection (i.e. user A) with "sendonly" and the offer for the remote peer (i.e. user B) with "recvonly".
  (I am not using the Unity.PeerConnection class)

The problem i faced is, that for the local PeerConnection (user A), the IceConnectionState never switches to any other than "Checking", although the procedure of exchanging ice candidates has been done. If im not totally wrong, that means a pair of candidates could not be found successfully (yet) by the Ice Agent.
If that is the case, the local frames aren't beeing transmitted to the remote peer.

For the remote PeerConnection, the IceConnectionState changes as expected from "Checking", over "Connected" to "Completed".

I have managed once to get the state to "Connected" and "Completed" when initiating the WebRTC workflow for the local PeerConnection. This happened only once and I had some BreakPoints in my Code which makes me think I might have a timing problem although sending out generated IceCandidates and processing for this PeerConnection received IceCandidates should not be meaningful for a timing problem. I can't seem to reproduce this behavior again and fail to have a successfull WebRTC session ongoing.

The PeerConnection.Connected event fires for both, the local PeerConnection and the remote PeerConnection.

You got maybe any idea why the IceConnectionState doesn't change for me in the described case or maybe a logical mistake?

FYI: On the backend-side I use Kurento.NET which opens up the WebRTC endpoints for each user and generates the remote IceCandidates and answers for my offers I send out client-sided.
The information is then beeing sent to a Kurento Media Server over wss://.

[drejx]

Hi @sesd777 ,

Sorry for the late reply! One thing I'd like to highlight in your case, if user A and user B want to connect and see each other then both their peer connections should be set to sendrecv. But if you have more than 2 users, say user A, B and C then my setup is like this:

user A  sendrecv on user B
user A recv on user C

user B sendrecv on user C
user B recv on user A

user C sendrecv on user A
user C recv on user B

Of course the above doesn't show that the users don't really have a direct connection to each other, but rather everything is going through a TURN server (that is the way I have it).

In terms of why you don't get an IceConnection Completed event it might be similar to what I was getting. I'm working with a custom TURN implementation, but maybe you're running into the same issue with Kurento. For each new peer connection instance that I created (e.g. user A recv on user C), I also had to open a connection to the signaling server (wss://) and make a request to subscribe/connect/attach to the system. If I didn't do that I would get rejected by the webrtc layer (security failure) and it would result in a IceConnection Failed event.

Hope it helps, good luck!

[sesd777]

Hi @drejx ,

thank you for your answer.
I exactly do what you mentioned about the connection to the signalling server for each new peer connection.
And yea my goal is to have a session ongoing with 3+ users, I used the example of user A and B to keep it simple to explain my problem. I have tried aswell to use sendrecv in any case but the result is the same. The event sometimes fires but often it doesn't. Im pretty sure that setting it to "sendonly" for yourself and to "recvonly" for the remote peers is a correct way in combination of Kurento, which does the rest. In current dev state we use google's coTURN with custom configuration or numb viagenies to exclude possible missconfiguration in the coTURN setup.

For me it looks like that the internal IceAgent struggles to find a corresponding pair of ice candidates although I'm using several STUN/TURN servers and the appropriate ports open.

[drejx]

The only other thing I think could be an issue is if the PeerConnectionConfiguration is not configured to expected values, for example, to connect to a TURN server:

            PeerConnectionConfiguration peerConfig = new PeerConnectionConfiguration()
            {
                IceServers = iceServerList,
                BundlePolicy = BundlePolicy.Balanced,
                IceTransportType = IceTransportType.Relay,
                SdpSemantic = SdpSemantic.PlanB
            };

And double-check whether the required SDP format is PlanB or Unified.
Edit: you should really try to get your logs from Kurento as that will be invaluable information to figure out what is going wrong.

Best of luck!

[zhuangce]

@drejx Hello, I would like to ask whether the current release1.0 supports multi-player communication.I tried to build a three-person communication function. There are two peerconnection in my scene. I found that the islocalvideotrackEnabled of the peerconnection initialized first would become true.However, the second peerconnection that is initialized is always false after addlocalvideotrackasync(). Then the second peerconnection will not receive the remote video.So I would like to ask you how can I establish a three-person video communication

[djee-ms]

For C++ and C#, you need 2 peer connections per person. In release/1.0 and the corresponding NuGet packages, the peer connection only supports 1 audio track and 1 video track per connection. If you are using Unity, you need similarly 2 distinct peer connection components. And in both cases each peer connection needs its own signaling. Basically you have to duplicate everything.

[zhuangce]

@djee-ms yes ,i'm using Unity ，the problem is the second initialized peerconnection seems like can't add a localvideotrack , its islocalvideotrackEnabled() property log false all the time ,by the way the perconnection 's isconnected property is true . so how can i send my local video to another two peerconnection ?

[djee-ms]

Ah right I just realized that won't work. The first peer connection will open the webcam, so the second one won't be able to access it. There is currently no solution for that, because the webcam is always opened with exclusive access.

One possible but very tedious workaround is to open and manage the webcam yourself, but this has several limitations:

• You need to use master, because you need external video track sources to inject the video frames of the webcam into WebRTC
• You need to rewrite some C++ track interfacing and C# wrapper
• You need to manage all the frame capture etc. by yourself
• You optionally need to rewrite a custom video track component for Unity

So this is a lot of work and I wouldn't recommend it.

Normally the typical way to support multiple peers (>2) is to use a media server, which handles duplicating the video feed to all participants. But that's a larger setup. That avoids having a network mesh topology though (1 connection per pair of participants), and replace it with a star topology.

Another way would be to separate the video track from its source, and allow a single source (webcam) to be shared by multiple tracks. This is what we already do for external video tracks in master, and we could extend it to the local video track opened from the webcam. Please feel free to open a new issue if you think that you need this feature, so we can consider it for planning.

[zhuangce]

that's exactly what i want to know , help a lot ,thank

[zhuangce]

@djee-ms yes , i'll open a new issue later ,seem's like sharing the webcam video source by multiple tracks is avaliable solution to me ,i'll try it later