Web RTC client and signal path in Home Assistant

[allenporter]

I would like to support the new Nest Battery Camera and Nest Battery Doorbell in Home Assistant. The camera differs from previous nest cameras in that instead of using an RTSP stream, only supports Web RTC.

Background

A typical WebRTC request flow might look something like this:

• Client: In the client in javascript start a connection via RTCPeerConnection and ask it for an SDP Offer
• Signaling: Pass the SDP Offer via a Signaling channel to the remote camera device, we’ll call the Peer
• Peer: Responds to the SDP Offer with an SDP Answer
• Client: Attaches an RTCSessionDescription with the SDP Answer to the RTCPeerConnection.
• Playback: As remote video tracks become available, the client can start receiving the video stream.

The Nest SDM API for webrtc camera streams basically works like this:

• Send a GenerateWebRtcStream RPC with an input offerSdp.
• The response contains an answerSdp.
• However, access to the stream expires after 5 minutes, and needs to be restarted. Yay, security! … but a little more complexity to manage.
• I don’t see a way to support an image thumbnail preview for what it’s worth. Presumably this saves battery.

Proposal

The high level idea is:

• Add Client support in frontend
• Add Signaling support in core via camera for components to implement. This is effectively just a pass through of the offer and answer to the Peer cloud or local camera device.
• Continue to use HLS for RTSP camera streams.
  • I do not have a strong stance on building a Peer in core as a WebRTC to RTSP bridge in the future, or deciding if such a bridge should live in stream.
  • A component could implement a signal to a remote service that acts as a bridge if desired, however, with this proposal.

Considerations

There are a few considerations to think through in more detail:

• The client or card needs to know that a camera supports WebRTC to use a separate different way to start the stream and connect to the remote peer
  • Should this be a new SUPPORT_WEBRTC or an additional property on the camera entity? We’re changing the fundamental definition of a stream here by breaking it down into separable pieces, and no longer having a stream source url.
• camera websocket: A new command that accepts the offer and returns the `answer
• camera entity model: could have a new method like async webrtc_offer(offer_sdp: str) -> answer_sdp: str
• There are likely fronted considerations, and i'm newer to this code base, so appreciate any early thoughts here
• As a result of the client driven playback, there would be no support for recording a WebRTC stream in core with this proposal.
• Browser compatibility: http://iswebrtcreadyyet.com/ says Safari is “yellow” and Safair iOS is “red” and recommends using an adapter javascript library. One worry is that client side support for WebRTC may be a large undertaking.
• As an initial simple approach for handling stream expiration: we may just get away with reconnecting to the stream on an error in the client.
• Is there another component that would immediately want to use WebRTC if it were available as a second use case to ensure we’ve built for the right requirements?
• There is a popular custom component https://github.com/AlexxIT/WebRTC that people use for RTSP streams. It has its own custom Client card, Signaling websocket, and Peer server that it runs as a separate binary. Not sure if there are any special considerations we should give for that custom component, but maybe I should get input from the author as a nice thing to do. Perhaps they'd be interested in contributing and /or have relevant experience to leverage.

I'd love thoughts from @uvjustin and @hunterjm in particular.

[allenporter]

@AlexxIT as the author of https://github.com/AlexxIT/WebRTC I am curious if you have any thoughts or want to weigh in.

[allenporter]

Implementation alternative: Preserve the notion of web rtc being a stream, and revamp all the assumptions about a stream and its interface with cameras and interface between frontend and core.

[balloob]

I don't think it's necessary to go this complicated. And it also makes the project a lot bigger. WebRTC is a beast, and browsers are good at it, we should let them deal with it.

[balloob]

I don't foresee any issues with viewing it in the browsers which support WebRTC. As long as we can establish the connection, we should be good. For iOS, we can have it fallback to open the stream natively in the app, overlaying the whole webview.

Recording not being part of it is also fine. Step 1 is to just view it.

I think that instead of a SUPPORT flag, we should probably add a new stream_type property. We can both the backend and the frontend treat such cameras differently, and that way we ensure it's mutually exclusive. We don't want to have a camera signal rtsp + webrtc and then not know what to do.

[hunterjm]

When I was initially building stream I was looking at using WebRTC for implementation. Specifically I played around with the aiortc library. That is how I discovered the pyav ffmpeg bindings. Implementing HLS turned out to be much simpler, and I never revisited it. aiortc makes a lot of assumptions and specifically always transcodes video to support multiple bitrates and respond to latency and adjustment requests from the client. That being said, it's dependencies are broken out pretty well with aioice to handle offers and establishing the connection, and pylibsrtp which provides python bindings for libsrtp to handle ecnrypt/decrypt of Secure Real-time Transport Protocol (SRTP) packets.

It's honestly a lot to bite off, and has it's limitations. That being said, if you want Home Assistant core as a proxy to Nest it's honestly probably not much of a stretch to repackage the RTSP (or other stream source) packets coming out of stream as RTP packets sent over a WebRTC connection. You would have to do that anyway to the packets coming from Google in your proxy.

[allenporter]

Talked through a bit with @hunterjm, @balloob, @uvjustin over discord and wanted to summarize my take-aways:

• Having a client side player and a minimal signal path that routes through core seems reasonable.
• Definitely makes sense to keep separate from stream
• We would strongly lead towards not having a RTSP bridge inside core. However, sharing this proposed client and signal path for other integrations that use a separate server for that seems totally fine. Perhaps we can chat about how to align this wfor the custom WebRTC component.
• We talked about options of also having the signal path be totally client side -- it could be in the nest example, but the integration details would also need to live in the frontend. Having a signal endpoint in camera that talks to the integration seems pretty reasonable.

Feel free to chime in if i didn't capture an important point.

Proposal

Elaborating on the above proposal in more detail:

• CameraEntity will add a new property exposing the protocol / type / format of the stream.
• The frontend will build an offer by creating a RTCPeerConnection and invoking createOffer()
• The offer is passed to a new websocket endpoint exposed in the camera component:

  type: camera/web_rtc_offer
  entity_id: string
  offer: string
  

  The API response is an answer
• To get the answer, the web_rtc_offer API implementation will invoke the CameraEnttiy with a new method called async def async_handle_web_rtc_offer(self, offer: string) -> string that returns an answer
• From there, the frontend handles everything else. It first creates a RTCSessionDescription from the answer as the remote description. Then adds remote tracks as they become available, etc.

I'd like to see if the mjpeg vs hls logic in frontend gets simplified by adding a stream type. Also, I know we've done some amount of cleanup of format which is unsed in the past, so I want to revisit that again to the reduce the number of terms we have.

[AlexxIT]

If the camera gives back WebRTC natively - that's great. The Hass is only needed for a secure SDP exchange. But it is not clear whether such cameras can bypass any NAT. My component can bypass some NATs, but not all. If NabuCasa is able to provide its users with a TURN server - that could be great.

My component uses a binary to convert the RTSP stream into a WebRTC stream. SDP is transmitted via Hass authorization, and the video stream is transmitted directly between the browser and the binary. Therefore, when working externally through Nabu Casa - the video stream does not load the server NC, this is a plus.

My component is able to transmit not only WebRTC, but also MSE within a single connection. Depending on the capabilities of the browser.

I planned to develop it further - to no delay snapshots, etc. But for now I am busy with other projects. Mostly Xiaomi Gateway 3...

[allenporter]

Thanks for weighing in @AlexxIT -- I'll proceed here and we can keep looking for ways to share notes and approaches so hopefully things can all play nicely together. It was helpful to read how your component works for informing an approach to support nest cams.

[rbray89]

I'll add another note, that libnice can be used (as used in janus gateway) to determine what IP to feed the stream request. I used a modified Janus Gateway instance to utilize UPNP (set a boolean in the libnice API) to successfully forward the UDP port.

A Janus Gateway addon to provide the webRTC server for camera streams should be pretty easy to throw together.