Add back option to use Frigate-native WebRTC support

[felipecrs]

• This partially reverts fix: Fix live streaming in HA >= 2024.11 #774

• To be merged after Correctly handle cameras that are not streaming #782

• Closes Improve support for HA 2024.11.0 WebRTC streaming #759

[felipecrs]

This is what I realized:

1. When using WebRTC native, indeed HA's go2rtc is not used to relay the RTSP into WebRTC.

   I was able to confirm this by enabling go2rtc.debug_ui in HA's configuration.yaml and testing WebRTC native (this PR) vs RTSP only. When no WebRTC native, a new stream is created within HA's go2rtc when I attempt to stream the camera. Otherwise, no stream is created there.

   This is proof that implementing WebRTC native is more efficient than RTSP only. (cc @OnFreund)

2. Documentation says the integration must implement async_handle_async_webrtc_offer and async_on_webrtc_candidate and have CameraEntityFeature.STREAM for native WebRTC to work.

   According to my tests, this is not true:

   a. async_on_webrtc_candidate does not need to be implemented.
   b. _attr_frontend_stream_type MUST be set to StreamType.WEB_RTC

   Otherwise, frontend fails with:

   {
     "code": "webrtc_get_client_config_failed",
     "message": "Camera does not support WebRTC, frontend_stream_type=hls"
   }

   But this comment from @edenhaus says that the integration was not supposed to manipulate this attribute.

3. This part of the documentation says:

   By implementing the WebRTC methods, the frontend assumes that the camera supports only WebRTC and therefore will not fallbac to HLS.

   I have found this not to be true, because I am able to cast my camera to my Chromecast devices (which is surely using HLS given the 10 seconds delay) even when the camera is set to WebRTC native.

@edenhaus, I'd like to kindly ask you to shed some light on 2 and 3, would you? PS: I had asked for your feedback here, but feel free to disregard that one and reply to here only.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 100.00%. Comparing base (72988b6) to head (0158825).
Report is 19 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff            @@
##            master      #784   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           17        17           
  Lines         2048      2080   +32     
=========================================
+ Hits          2048      2080   +32     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[felipecrs]

Ok, I found these PRs:

• Use camera_capabilities instead frontend_stream_type home-assistant/core#130604
• Deprecate camera frontend_stream_type home-assistant/core#130932

I believe they will fix the second bullet. I will test and update the PR once a new HA beta is released with them.

EDIT: just found your blog post btw: https://developers.home-assistant.io/blog/2024/11/26/camera-deprecations/

EDIT 2: the second bullet is actually fixed by this exact change:

home-assistant/core@154282f#diff-75655c0eec1c3e736cad1bdb5627100a4595ece9accc391b5c85343bb998594fR598-R603

Such commit not only deprecates the attribute but also fixes the method to return the proper frontend_stream_type when the WebRTC methods are implemented.

This means I'll need to keep setting the attribute until the new version of HA is out.

[felipecrs]

@edenhaus, this section of the code is according to the documentation (third bullet):

https://github.com/home-assistant/core/blob/ce20670d844373d49a64525ee59abfc8507ccd2e/homeassistant/components/camera/__init__.py#L927-L929

I suppose the HLS player still plays my camera entity because it doesn't take frontend_stream_types into consideration. It just gets stream_source and tries to play it, and given Frigate integration implements stream_source, it works.

I believe this logic can be improved. Maybe somehow detecting if stream_source would yield any result?

If you can manage to do so, this PR (of mine) can be reverted:

• Allow to cast in HLS format when using WebRTC home-assistant/core#80646

[felipecrs]

Another potential reason why the current approach is inefficient:

The Frigate integration will provide an RTSP link with ?mp4 (transparently), meaning Frigate's go2rtc will serve AAC as the audio codec. However, for HA WebRTC, AAC cannot be used.

The HA integration will ask HA's go2rtc to transcode back AAC to OPUS:

https://github.com/home-assistant/core/blob/e04b6f0cd86c0490ff5a9c74b79b653de9a41a60/homeassistant/components/go2rtc/__init__.py#L257

This by itself will consume some CPU %. You can multiply it to the number of cameras streaming at the same time.

[felipecrs]

@dermotduffy raised this concern:

What if the user doesn't have WebRTC properly configured in the Frigate settings?

https://docs.frigate.video/configuration/live/#webrtc-extra-configuration

[NickM-27]

we can check for that if we want to

[felipecrs]

The ideal would be HA to fallback to HLS in this case. I will try to provoke this situation and see what happens.

(despite the fact that HA documentation says camera entities with native WebRTC will not fallback to HLS)

[NickM-27]

Presumably if that doesn't work we can just return None in the webrtc functions if webrtc is not enabled, and it will function the same as it does today

[felipecrs]

if webrtc is not enabled

I could not think of a way to detect that. For example, there's no obvious setting to disable webrtc in go2rtc. Do you have any idea?

[NickM-27]

Just check if go2rtc -> webrtc -> candidates is set

[dermotduffy]

Presumably if that doesn't work we can just return None in the webrtc functions if webrtc is not enabled, and it will function the same as it does today

Is this correct? I thought it was the presence of those methods that mattered?

[NickM-27]

Just check if go2rtc -> webrtc -> candidates is set

This is interesting. It doesn't prove user has configured WebRTC for external access, but at least means internal access will work.

That is what the Frigate UI uses so I would suggest just doing the same

Also, what if we set the stream type to HLS and still implement the webrtc functions - would that work instead of creating new classes?

technically we are not supposed to change that value, but I believe for HA 2024.11 you have to set it.

[dermotduffy]

I don't think it's that complicated.

If we want to go down this path, maybe a new option is the simplest / least surprising and lets people always go back to "HA native" (more likely to be future-proof?) if something breaks:

• Option name something like "Use Frigate-native WebRTC support" as @OnFreund suggests.
• Option is disabled by default. Up for debate, but as I strongly suspect most people won't care, and this is the path least likely to break, this is my suggestion...
• Document the option in README.md to explain what it does and what the circumstances are that you may wish to enable it.
• Depending on the option we choose a different flavor the camera entity, where the only difference is the presence of the webrtc methods.
• ...
• Profit?

[NickM-27]

Agreed 👍

[felipecrs]

I don't think it's that complicated.

Unfortunately, it is: if you "tag" a camera as WebRTC-capable, Home Assistant will not fallback to HLS even if WebRTC doesn't work, and even if it provides everything HLS needs (stream_source). This is written in their documentation, meaning you can't have both at the same time. But this can probably be improved though, maybe it was just overlooked by HA core.

[felipecrs]

Just check if go2rtc -> webrtc -> candidates is set

@NickM-27, BTW, this isn't even needed if you are running Frigate through the add-on. Or if you are running with --network=host. So, I don't think we can rely on it.

[felipecrs]

I'm working to bring back the option.

[NickM-27]

BTW, this isn't even needed if you are running Frigate through the add-on

Isn't it? My understanding was when you run via the addon it just sets that value automatically.

And running as host is not recommended

[felipecrs]

Isn't it? My understanding was when you run via the addon it just sets that value automatically.

Yes, but it would not be part of the _frigate_config object (which is where the integration reads from), since it's only added to the /dev/shm/go2rtc.yaml.

[NickM-27]

Gotcha, we could probably add some field for that, but either way the manual option should work well for now

[felipecrs]

@dermotduffy, @NickM-27, here is my recommendations for the release notes:

Remove:

• fix: Fix live streaming in HA >= 2024.11 (fix: Fix live streaming in HA >= 2024.11 #774) @dermotduffy

Add:

• Fix live streaming in HA >= 2024.11 when the Frigate-native WebRTC option is not enabled (Add back option to use Frigate-native WebRTC support #784) @felipecrs
• Add support to stream birdseye through Frigate-native WebRTC (Add back option to use Frigate-native WebRTC support #784) @felipecrs
• Fix Camera APIs deprecated in 2024.12 (Add back option to use Frigate-native WebRTC support #784) @felipecrs

Special mention:

Frigate-native WebRTC

Home Assistant 2024.11 ships with support for streaming any camera entity through WebRTC by relaying it through go2rtc. As you probably know, Frigate also uses go2rtc to support WebRTC streaming. If you have WebRTC configured and working in Frigate, we recommend you tick the Use Frigate-native WebRTC support integration option in the CONFIGURE button.

When this option is enabled, Home Assistant will leverage Frigate's go2rtc to provide the WebRTC stream, rather than having to relay the stream into Home Assistant's own go2rtc, which consumes significantly more resources of your server (for example, Home Assistant's go2rtc will transcode AAC audio to OPUS).

[felipecrs]

@NickM-27 @dermotduffy this is fully tested in my own Home Assistant instance. Ready to go from my side.

[dermotduffy]

Thanks for the persistence @felipecrs !