More options for SDP and video settings

[eirikhollis]

Hi,

I'm currently initiating calls with the SDP i receive from the LocalSdpReadytoSend, but as I understand, it has already been set as local description at that time.
It would be beneficial if I could e.g. modify specific OPUS parameters as Max Average Bitrate, Max Capture Rate, Max Playback Rate and so forth. Same goes for video parameters such as resolution and framerate, but as I understand, that's already in the works. 👍

Maybe CreateOffer() could have an optional input variable for specific SDP options?

[djee-ms]

There are two concepts here, and I need to dig more into how they interact:

1. Parameters for the local video/audio capture, which the upcoming video profile changes are part of. This controls the capture only, and I think dictates the default resolution and framerate on connection start before any dynamic adjustment is made.

2. Parameters for the SDP negotiations for the video/audio transport through the peer connection and the bandwidth management.

SDP packet filtering and adding more options work on the second one. As I understand these constraints are maximum values to optimize network transport.

But for a scenario where you want to reduce the CPU load and/or bandwidth by limiting the video resolution you'd rather constrain directly the video capture module than the SDP offer, so that frames are directly captured in the format you want, and no transformation is needed. Otherwise capturing at 4K to transport at 640 x 480 will waste local CPU cycles to downscale the video stream before sending.

In short you need I think 2 things:

• Tell webrtc what are the limits it should work within (SDP)
• Tell the video/audio capture module what format to record in, which hopefully match the transport format so that no transform is needed, and at least is no higher than the max spec you expect to transport.

Does that sound correct?

[eirikhollis]

That is correct, and a much better summary of the feature than I managed to write.

As an example of an use case for resolution and framerate:
In my application, a specific setup part between two peers is that I need to specify the resolution and framerate that I intend to transmit, such that the receiver can use those values for internal UI setup or other video related things. Since I'm not in control of the resolution, nor framerate, I have to guess/hardcode these values for now to something generic, say 1280x720x30.

Of course, the CPU utilization savings are a big part of why I would like control of the resolution, but there could also be other reasons, such as the one I mentioned above

[djee-ms]

Dumping some info from today's investigation:

• The OPUS-specific RFC is here : https://tools.ietf.org/html/draft-spittka-payload-rtp-opus-03#section-6.2

• The OPUS-specific constants, and all others as well, are in mediaconstants.cc.

• The SDP parsing of constants for all codecs is done in webrtcsdp.cc

• Encoder code, with actually supported parameters, is in audio_encoder_opus.cc. But it doesn't use the media constants, it's just hard-coded strings.

It seems a priori from those info that it would be possible to support some SDP filtering for OPUS, possibly even a more generic format for other codecs.

[eirikhollis]

Not sure about if it has any effect on performance, it could also be interesting to be able to modify lines 275-276 and 291-291 in peer_connection.cpp

options.offer_to_receive_audio = true;
options.offer_to_receive_video = true;

Now this is a special case for my implementation, but I only either send OR receive video/audio on each peer connection, so previously to this plugin, I modified offer_to_receive_audio/video to false depending on the connection. I also changed sendrecv in the sdp to sendonly or recvonly depending on the use-case.

More information on sendrecv

[djee-ms]

Re-opening after GitHub's auto-close.

[djee-ms]

Closing this issue, now fixed with video capture format available via media constraints (a2a9855). Opened #75 for the offer_to_receive_xxx options, which are deprecated anyway.