Card stops working intermittently: Unknown error #568

felipecrs · 2022-05-10T14:07:44Z

I'm using the current latest v4.0.0. Nothing shows in the browser logs. Happens frequently, but I still can't determine why.

chrome_YpJe2qR4WB.mp4

This didn't happen in v3.

This is the configuration of one of my cards, but the problem happens to all of them:

- type: custom:frigate-card
    view:
      default: live
      update_force: false
    menu:
      buttons:
        frigate:
          enabled: false
        snapshots:
          enabled: false
        image:
          enabled: false
      style: outside
      position: top
    live:
      preload: false
      controls:
        next_previous:
          style: none
        title:
          mode: none
      lazy_load: true
      lazy_unload: all
      auto_unmute: never
    dimensions:
      aspect_ratio_mode: static
      aspect_ratio: '16:9'
    event_viewer:
      auto_play: never
    elements:
      - type: custom:frigate-card-conditional
        conditions:
          view:
            - live
          camera:
            - camera.rua
        elements:
          - type: icon
            icon: mdi:arrow-up
            style:
              background: rgba(255, 255, 255, 0.75)
              border-radius: 5px
              right: 25px
              bottom: 115px
            tap_action:
              action: call-service
              service: onvif.ptz
              service_data:
                entity_id: camera.rua_onvif
                tilt: UP
                move_mode: ContinuousMove
                speed: 0.05
                continuous_duration: 0.05
          - type: icon
            icon: mdi:arrow-down
            style:
              background: rgba(255, 255, 255, 0.75)
              border-radius: 5px
              right: 25px
              bottom: 65px
            tap_action:
              action: call-service
              service: onvif.ptz
              service_data:
                entity_id: camera.rua_onvif
                tilt: DOWN
                move_mode: ContinuousMove
                speed: 0.05
                continuous_duration: 0.05
          - type: icon
            icon: mdi:arrow-left
            style:
              background: rgba(255, 255, 255, 0.75)
              border-radius: 5px
              right: 50px
              bottom: 90px
            tap_action:
              action: call-service
              service: onvif.ptz
              service_data:
                entity_id: camera.rua_onvif
                pan: LEFT
                move_mode: ContinuousMove
                speed: 0.05
                continuous_duration: 0.05
          - type: icon
            icon: mdi:arrow-right
            style:
              background: rgba(255, 255, 255, 0.75)
              border-radius: 5px
              right: 0px
              bottom: 90px
            tap_action:
              action: call-service
              service: onvif.ptz
              service_data:
                entity_id: camera.rua_onvif
                pan: RIGHT
                move_mode: ContinuousMove
                speed: 0.05
                continuous_duration: 0.05
      - type: custom:frigate-card-conditional
        conditions:
          view:
            - live
          camera:
            - camera.rua
        elements:
          - type: custom:frigate-card-menu-submenu-select
            entity: select.camera_da_rua_night_vision
    cameras:
      - camera_entity: camera.rua
        live_provider: frigate-jsmpeg
        id: camera.rua
        title: Rua
        camera_name: rua
      - camera_entity: camera.morro
        live_provider: frigate-jsmpeg
      - camera_entity: camera.portao
        live_provider: frigate-jsmpeg
      - camera_entity: camera.video_porteiro
        live_provider: ha

dermotduffy · 2022-05-10T14:18:35Z

Thanks @felipecrs .

Ever see it in a view other than live? You are using multiple live providers, does it show up on providers or only 1? (If you happened to notice!)

felipecrs · 2022-05-10T14:27:28Z

It happens in clips view as well.

By the way I noticed a bunch of errors in the Hass Core logs:

I'm not sure what's the best way to dump these logs (and btw, if Frigate integration supported Diagnostics this would be a little easier).

felipecrs · 2022-05-10T14:28:38Z

Here is one of them:

Logger: homeassistant.components.websocket_api.http.connection
Source: custom_components/frigate/media_source.py:685
Integration: Home Assistant WebSocket API (documentation, issues)
First occurred: May 9, 2022, 11:53:12 AM (374 occurrences)
Last logged: 11:21:45 AM

[139964827719376] Error handling message: Unknown error (unknown_error)
[139964821966464] Error handling message: Unknown error (unknown_error)
[139965014090464] Error handling message: Unknown error (unknown_error)
[139962666910048] Error handling message: Unknown error (unknown_error)
[139964532992512] Error handling message: Unknown error (unknown_error)
Traceback (most recent call last):
  File "/config/custom_components/frigate/api.py", line 140, in api_wrapper
    response = await self._session.get(
  File "/usr/local/lib/python3.9/site-packages/aiohttp/client.py", line 559, in _request
    await resp.start(conn)
  File "/usr/local/lib/python3.9/site-packages/aiohttp/client_reqrep.py", line 898, in start
    message, payload = await protocol.read()  # type: ignore[union-attr]
  File "/usr/local/lib/python3.9/site-packages/aiohttp/streams.py", line 616, in read
    await self._waiter
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/config/custom_components/frigate/api.py", line 154, in api_wrapper
    await self._session.post(url, headers=headers, json=data)
  File "/usr/local/lib/python3.9/site-packages/async_timeout/__init__.py", line 129, in __aexit__
    self._do_exit(exc_type)
  File "/usr/local/lib/python3.9/site-packages/async_timeout/__init__.py", line 212, in _do_exit
    raise asyncio.TimeoutError
asyncio.exceptions.TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/config/custom_components/frigate/media_source.py", line 681, in _get_event_summary_data
    summary_data = await self._get_client(identifier).async_get_event_summary(
  File "/config/custom_components/frigate/api.py", line 102, in async_get_event_summary
    await self.api_wrapper(
  File "/config/custom_components/frigate/api.py", line 162, in api_wrapper
    raise FrigateApiClientError from exc
custom_components.frigate.api.FrigateApiClientError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/decorators.py", line 27, in _handle_async_response
    await func(hass, connection, msg)
  File "/usr/src/homeassistant/homeassistant/components/media_source/__init__.py", line 152, in websocket_browse_media
    media = await async_browse_media(hass, msg.get("media_content_id", ""))
  File "/usr/src/homeassistant/homeassistant/components/media_source/__init__.py", line 111, in async_browse_media
    item = await _get_media_item(hass, media_content_id).async_browse()
  File "/usr/src/homeassistant/homeassistant/components/media_source/models.py", line 85, in async_browse
    return await self.async_media_source().async_browse_media(self)
  File "/config/custom_components/frigate/media_source.py", line 653, in async_browse_media
    await self._get_event_summary_data(identifier), identifier, events
  File "/config/custom_components/frigate/media_source.py", line 685, in _get_event_summary_data
    raise MediaSourceError from exc
homeassistant.components.media_source.error.MediaSourceError

dermotduffy · 2022-05-10T15:01:23Z

Looks like a communication issue between HA and the Frigate backend, which propagated to a failed media source error, which propagated into the card.

Not sure why this wouldn't have happened with v3 though. Did you change your Frigate backend version recently?

felipecrs · 2022-05-10T15:03:20Z

No, I didn't change anything else, except applying HA Core patch upgrade, I think.

I didn't reboot my server since a while, I will reboot it now and monitor too.

dermotduffy · 2022-05-10T15:04:20Z

Actually, I think the v4 card is probably fetching the events to show as thumbnails, but you don't have thumbnails enabled -- so that fetch does ~nothing useful. And when that fetch fails, it causes this error for you. I can try to optimize that away ... [and off the top of my head the v3 probably didn't behave this way]

(But still seems like there's something unrelated going on between your HA and your Frigate instance)

felipecrs · 2022-05-10T15:05:30Z

Ok, error remains happening after rebooting the server, just to confirm.

dermotduffy · 2022-05-10T15:15:03Z

Anything hints in your Frigate backend logs?

dermotduffy · 2022-05-10T15:17:00Z

BTW: Because of the behavior I described above, if you built your v4 before I fixed #557 , this problem may have been massively exasperated (as fetching was incorrectly continuous, instead of one-off).

felipecrs · 2022-05-10T15:19:19Z

Ok, but my card is up-to-date with the release-4.0.0 tree, including my own PR which was the last to be merged.

felipecrs · 2022-05-10T15:20:19Z

I will try to revert to 3.0.0 and see what happens, I will let you know of the result.

felipecrs · 2022-05-10T15:22:14Z

Anything hints in your Frigate backend logs?

No, nothing useful there. Only the regular WebSockets initiated and terminated messages.

felipecrs · 2022-05-10T16:01:07Z

I wasn't able to reproduce the issue in v3 (but testing it wasn't very sharp, as my card configuration was made for v4, what I did was to create another card with 3 cameras lacking a lot of the configuration from my real ones)
After rebuilding and upgrading again to the v4 tree (Fix card breakage on Nest hub devices #569), I yet could not reproduce the issue.

Let me monitor for a while.

felipecrs · 2022-05-10T18:12:30Z

It just happened again. I noticed that it happens not only during initialization. Right now, it happened around 3 seconds after the camera was already being shown.

felipecrs · 2022-05-10T18:13:48Z

I think it has no relationship with cameras being Frigate or not (because I do mix them, but the third card in the above screenshot only mixes Frigate cameras).

felipecrs · 2022-05-11T13:39:18Z

Ok, the issue came from frequently to rare. I don't know what has changed, and I can't guarantee anymore that this was introduced by v4.

I still see tons of Unknown error in my HA Core logs, related to the Frigate WebSockets API, but, they are not causing my card to crash (it seems).

So, I'm closing this. Thank you very much for the support so far, if I find something useful I will let you know.

felipecrs · 2022-05-11T15:53:32Z

Out of the blue this started to plague me again:

chrome_t1KPSoxcVs.mp4

I have the feeling that this has something to do with ongoing detections/clips.

felipecrs · 2022-05-11T16:06:09Z

Oh, I found a good way to test.

I was using HA 2022.5.3, and I restored the HA Core backup of 2022.5.2. This also restored my Lovelace dashboard, which in turn was before the Frigate card 4.0.0.

This way, I could revert to 3.0.0 keeping all my cards untouched.

I am testing now with v3.

felipecrs · 2022-05-11T16:14:19Z

Just to mention, restoring the backup caused a big side effect for me:

Restore of core backup deleted entries in config folder home-assistant/core#68198

The correct thing to do was to restore the backup taken by the Google Drive add-on. That one restored everything, including my /config folder.

felipecrs · 2022-05-11T16:37:18Z

@dermotduffy, I think I was able to narrow the issue a bit further:

https://1drv.ms/v/s!AntpCaY1YjnMiukyVoZUxB_tmqUm7A?e=UCFE1S

(The video is too big to post on GitHub)

dermotduffy · 2022-05-14T02:45:38Z

Theory: The card is spamming the server with requests, causing the Frigate backend to get overloaded, time out on some of those requests -- causing this error.

dermotduffy · 2022-05-14T03:55:17Z

@felipecrs Would you mind re-testing with the merged #587 ? I hope/suspect this will help you!

felipecrs · 2022-05-14T16:43:51Z

@dermotduffy thank you so much! But unfortunately, it did not solve the issue:

chrome_zKby95dqnH.mp4

(Jump to 0:35)

Now I will try to update to the latest Frigate integration that was just released.

felipecrs · 2022-05-14T16:45:05Z

I have another theory:

Now the card fetches thumbnails (which didn't happen before)
Maybe there is a bug in the Frigate backend which affects my environment (but not yours), and it was related to the thumbnails API, and since now the card is pulling from it, it triggers the error.

dermotduffy · 2022-05-14T17:02:30Z

Thanks @felipecrs . Ah well, we'll get there!

In your configuration, the card should no longer fetch thumbnails in the live view (since you do not enable thumbnails in your config). I notice in your video shows the problem in timeline view (which does need to fetch events, obviously) -- so at least this might be resolved for you in live view?

Your theory could be right. Is your Frigate machine under heavy load perhaps? Does browsing the HA media browser ever fail for you?

I'll do a little research later on trying to reduce the number of requests the timeline is showing, but that'll be harder since events are kind of the whole point. To experiment, one simple way to cut the number of event requests in half, would be to do this:

timeline:
  media: clips

(By default it fetches events that are clips & snapshots, so limiting it to one of those instantly cuts the number of fetches necessary by 50%).

felipecrs · 2022-05-14T17:10:27Z

@dermotduffy thanks a lot for the insights, but please don't do anything yet. I'm testing the new Frigate integration, and I already see a different behavior:

Before, in the timeline view, nothing ever showed up for me (I was about to send you a Discord message asking if that was expected). Immediately after upgrading the integration, I now get proper timeline views:

And yet, no errors.

dermotduffy · 2022-05-14T17:22:51Z

Yes, that is expected. The new integration has blakeblackshear/frigate-hass-integration#225 which is required for the timeline to work (I just hadn't done an integration build, since the card code is still very untested!).

OK -- I'll hold off until you tell me.

felipecrs · 2022-05-14T17:36:29Z

@dermotduffy I'm about to claim everything is good now. No errors in HA core logs too. If nothing shows up by the end of day, I will close the issue.

felipecrs · 2022-05-14T17:43:13Z

And just to answer some questions:

so at least this might be resolved for you in live view?

I haven't observed the issue happening in live view indeed, but my test (before upgrading the Frigate integration) was very shallow.

Is your Frigate machine under heavy load perhaps?

Yes, the machine is under a good load. I host everything on it through HassOS, and Frigate usually takes 40-60% of my 4-core J4125 CPU. The total load is usually at 70%:

I haven't yet determined the reason behind the 90-100% spikes, though.

Does browsing the HA media browser ever fail for you?

Never noticed it.

felipecrs · 2022-05-14T18:56:43Z

Ok, the issue didn't happen since I upgraded the Frigate integration from v2.2.2 to v2.3.

I'm closing the issue, if it happens again, I will come back.

I am very grateful about all the care you put in both the Frigate Card and the integration.

satyambaba · 2022-06-06T17:36:13Z

@dermotduffy, I am facing the same issue even on Frigate Integration v2.3 (Frigate Card - v3.0.0 and Frigate Addon - Frigate NVR v3.1). I am on HA Blue (HA Core 2022.6.2, Supervisor 2022.05.3, OS 8.1).

dermotduffy · 2022-06-07T00:24:25Z

Hi @satyambaba, I'm not sure that's the same thing. Felipe's error was showing up as "unknown", whereas yours is labeled as a media source issue.

Does the HA Media Browser work correctly for you for Frigate? Do you have any exceptions in your Home Assistant logs, or anything on your Javascript console?

satyambaba · 2022-06-07T00:47:04Z

Hi @satyambaba, I'm not sure that's the same thing. Felipe's error was showing up as "unknown", whereas yours is labeled as a media source issue.

Does the HA Media Browser work correctly for you for Frigate? Do you have any exceptions in your Home Assistant logs, or anything on your Javascript console?

Hi @dermotduffy, thanks a lot for your response. I can browse frigate clips and recordings using the inbuilt media browser. I don't have any exception in HA logs or in the JS console. Please see the attached screenshots.

dermotduffy · 2022-06-07T00:50:49Z

@satyambaba What version number do you see when you navigate to your Frigate UI, then click debug on the left sidebar? You should see a version number near the top of the screen.

satyambaba · 2022-06-07T00:53:54Z

@satyambaba What version number do you see when you navigate to your Frigate UI, then click debug on the left sidebar? You should see a version number near the top of the screen.

@dermotduffy - I see 0.10.1-83481AF. I'm not sure if this is the version number you are looking for?

dermotduffy · 2022-06-07T01:01:49Z

Bit surprised you're seeing mp4 files in the media browser like that, thought you might be on a much older Frigate instance. Something feels wrong here :-)

Can you navigate in HA to Settings -> Devices & Services, then click through your Frigate entry, choose 'Devices' choose any arbitrary Frigate device that is other than the one named Frigate then take a screenshot? It should look something like this:

satyambaba · 2022-06-07T01:10:53Z

Bit surprised you're seeing mp4 files in the media browser like that, thought you might be on a much older Frigate instance. Something feels wrong here :-)

Can you navigate in HA to Settings -> Devices & Services, then click through your Frigate entry, choose 'Devices' choose any arbitrary Frigate device that is other than the one named Frigate then take a screenshot? It should look something like this:

While I was navigating to take the suggested screenshot, I noticed that the frigate integration was disabled. The moment I enabled that, the error vanished. Sorry for the trouble, it was an oversight at my end. However, now I'm wondering that how the frigate was working fine (including capturing events, recording, etc.) while the integration was disabled.

dermotduffy · 2022-06-07T01:12:16Z

While I was navigating to take the suggested screenshot, I noticed that the frigate integration was disabled.

Aha!

The moment I enabled that, the error vanished. Sorry for the trouble, it was an oversight at my end. However, now I'm wondering that how the frigate was working fine (including capturing events, recording, etc.) while the integration was disabled.

The integration only does the glue between the Frigate server (which captures events/records) and Home Assistant, so that is as expected.

satyambaba · 2022-06-07T01:13:42Z

While I was navigating to take the suggested screenshot, I noticed that the frigate integration was disabled.

Aha!

The moment I enabled that, the error vanished. Sorry for the trouble, it was an oversight at my end. However, now I'm wondering that how the frigate was working fine (including capturing events, recording, etc.) while the integration was disabled.

The integration only does the glue between the Frigate server (which captures events/records) and Home Assistant, so that is as expected.

Thanks for the clarification, and your help. I really appreciate it! Thanks a lot for this awesome integration.

felipecrs added the bug Something isn't working label May 10, 2022

dermotduffy added this to the v4.0.0 milestone May 10, 2022

felipecrs closed this as completed May 11, 2022

felipecrs changed the title ~~Card stops working intermittently~~ Card stops working intermittently: Unknown error May 11, 2022

felipecrs reopened this May 11, 2022

dermotduffy mentioned this issue May 14, 2022

Reduce thumbnail fetch frequency to prevent Frigate request timeouts #587

Merged

felipecrs closed this as completed May 14, 2022

github-actions bot locked and limited conversation to collaborators Oct 7, 2024

Card stops working intermittently: Unknown error #568

Card stops working intermittently: Unknown error #568

Comments

felipecrs commented May 10, 2022 • edited Loading

dermotduffy commented May 10, 2022

felipecrs commented May 10, 2022

felipecrs commented May 10, 2022

dermotduffy commented May 10, 2022

felipecrs commented May 10, 2022

dermotduffy commented May 10, 2022 • edited Loading

felipecrs commented May 10, 2022

dermotduffy commented May 10, 2022

dermotduffy commented May 10, 2022

felipecrs commented May 10, 2022

felipecrs commented May 10, 2022

felipecrs commented May 10, 2022

felipecrs commented May 10, 2022

felipecrs commented May 10, 2022 • edited Loading

felipecrs commented May 10, 2022

felipecrs commented May 11, 2022

felipecrs commented May 11, 2022

felipecrs commented May 11, 2022 • edited Loading

felipecrs commented May 11, 2022 • edited Loading

felipecrs commented May 11, 2022 • edited Loading

dermotduffy commented May 14, 2022

dermotduffy commented May 14, 2022

felipecrs commented May 14, 2022

felipecrs commented May 14, 2022

dermotduffy commented May 14, 2022

felipecrs commented May 14, 2022 • edited Loading

dermotduffy commented May 14, 2022

felipecrs commented May 14, 2022

felipecrs commented May 14, 2022 • edited Loading

felipecrs commented May 14, 2022

satyambaba commented Jun 6, 2022 • edited Loading

dermotduffy commented Jun 7, 2022

satyambaba commented Jun 7, 2022

dermotduffy commented Jun 7, 2022

satyambaba commented Jun 7, 2022

dermotduffy commented Jun 7, 2022

satyambaba commented Jun 7, 2022

dermotduffy commented Jun 7, 2022

satyambaba commented Jun 7, 2022

felipecrs commented May 10, 2022 •

edited

Loading

dermotduffy commented May 10, 2022 •

edited

Loading

felipecrs commented May 10, 2022 •

edited

Loading

felipecrs commented May 11, 2022 •

edited

Loading

felipecrs commented May 11, 2022 •

edited

Loading

felipecrs commented May 11, 2022 •

edited

Loading

felipecrs commented May 14, 2022 •

edited

Loading

felipecrs commented May 14, 2022 •

edited

Loading

satyambaba commented Jun 6, 2022 •

edited

Loading