Serialize amcrest snapshot commands and bump PyPI package to 1.2.4 #21664

pnbruckner · 2019-03-04T21:48:20Z

Description:

Attempting to send a snapshot command when a previous one hasn't finished will result in warnings and/or errors. This can happen when the camera picture is clicked on in the frontend, resulting in the thread that updates the thumbnail in the background every 10 seconds to sometimes collide with the thread that updates the large picture in the foreground quickly. An automation that calls the camera.snapshot service in yet another thread can make the situation worse. Fix by adding a thread lock to serialize snapshot commands. Also bump the amcrest package to 1.2.4 which fixes error handling in the command method and improves performance by reusing requests sessions.

Related issue (if applicable): None

Pull request in home-assistant.io with documentation (if applicable): None

Example entry for `configuration.yaml` (if applicable):

No change to configuration.

Checklist:

The code change is tested and works locally.
Local tests pass with tox. Your PR cannot be merged unless tests pass
There is no commented out code in this PR.

If the code communicates with devices, web services, or third-party tools:

New or updated dependencies have been added to requirements_all.txt by running script/gen_requirements_all.py.

Attempting to send a snapshot command when a previous one hasn't finished will result in warnings and/or errors. This can happen when the camera picture is clicked on in the frontend, resulting in the thread that updates the thumbnail in the background every 10 seconds to sometimes collide with the thread that updates the large picture in the foreground quickly. An automation that calls the camera.snapshot service in yet another thread can make the situation worse. Fix by adding a thread lock to serialize snapshot commands. Also bump the amcrest package to 1.2.4 which fixes error handling in the command method and improves performance by reusing requests sessions.

amelchio · 2019-03-04T22:07:21Z

You deleted the part of the checklist that relates to the action that you forgot ...

Change looks good 👍

pnbruckner · 2019-03-04T22:55:36Z

@amelchio

You deleted the part of the checklist that relates to the action that you forgot ...

What part/action was that? I deleted everything I though wasn't applicable.

amelchio · 2019-03-05T05:14:07Z

There is (was) a whole section on new dependencies. You have to update requirements_all.txt as well for the Travis test to pass.

pnbruckner · 2019-03-05T13:53:01Z

Ah, ok. Sorry, I missed that. Should be good now. Thx!

amelchio

Looks good 👍

amelchio · 2019-03-05T15:59:35Z

homeassistant/components/amcrest/camera.py

+            try:
+                # Send the request to snap a picture and return raw jpg data
+                return self._camera.snapshot(channel=self._resolution).data
+            except (RequestException, ReadTimeoutError, ValueError) as error:


It's a shame the library does not handle these exceptions and raises its own instead.

Good idea. I'm working on the library, too, so I can look at doing that. Thanks!

MartinHjelmare · 2019-03-05T17:48:56Z

homeassistant/components/amcrest/camera.py

-        # Send the request to snap a picture and return raw jpg data
-        response = self._camera.snapshot(channel=self._resolution)
-        return response.data
+        with self._snapshot_lock:


We should avoid waiting on locks in our thread pool. We can switch to the async api and use an asyncio.Lock instead.

Oh. I agree that would be better but are you implying that we actually have a rule against locks?

Yeah, it's basically the same rule as for sleeping in the thread pool.

Hm. I think that dilutes the value of the thread pool. What if the lock was in the third party library? Maybe we should disallow I/O as well since it will sleep?

@pnbruckner I think this is the rewrite that Martin suggests:

async def async_camera_image(self): """Return a still image response from the camera.""" with self._snapshot_lock: try: # Send the request to snap a picture and return raw jpg data response = await hass.async_add_executor_job( self._camera.snapshot, self._resolution) return response.data except (RequestException, ReadTimeoutError, ValueError) as error: _LOGGER.error( 'Could not get camera image due to error %s', error) return None

(and then _snapshot_lock should be an asyncio.Lock())

Ok. So how do I go about this? Do I restore my branch and check in more changes? Do I open a new PR? Something else?

That should be a new PR.

@amelchio @MartinHjelmare

I need some help on this. This is what I came up with based on the suggested code above:

self._snapshot_lock = asyncio.Lock() async def async_camera_image(self): """Return a still image response from the camera.""" with await self._snapshot_lock: try: # Send the request to snap a picture and return raw jpg data _LOGGER.critical('async_camera_image start') response = await self.hass.async_add_executor_job( self._camera.snapshot, self._resolution) _LOGGER.critical('async_camera_image end: %i', len(response.data)) return response.data except (RequestException, ReadTimeoutError, ValueError) as error: _LOGGER.error( 'Could not get camera image due to error %s', error) return None

But it doesn't seem to be working. I added the logging temporarily to try and see what's going on. I also added logging to the self._camera.snapshot method. Here is what I'm seeing:

2019-03-05 17:53:52 CRITICAL (MainThread) [homeassistant.components.amcrest.camera] async_camera_image start 2019-03-05 17:53:52 CRITICAL (SyncWorker_1) [amcrest.snapshot] snapshot start 2019-03-05 17:53:53 CRITICAL (SyncWorker_1) [amcrest.snapshot] snapshot end 2019-03-05 17:54:00 CRITICAL (MainThread) [homeassistant.components.amcrest.camera] async_camera_image end: 158840 2019-03-05 17:54:00 CRITICAL (MainThread) [homeassistant.components.amcrest.camera] async_camera_image start 2019-03-05 17:54:00 CRITICAL (SyncWorker_7) [amcrest.snapshot] snapshot start 2019-03-05 17:54:01 CRITICAL (SyncWorker_7) [amcrest.snapshot] snapshot end 2019-03-05 17:54:09 CRITICAL (MainThread) [homeassistant.components.amcrest.camera] async_camera_image end: 159495 2019-03-05 17:54:09 CRITICAL (MainThread) [homeassistant.components.amcrest.camera] async_camera_image start 2019-03-05 17:54:09 CRITICAL (SyncWorker_2) [amcrest.snapshot] snapshot start 2019-03-05 17:54:09 CRITICAL (MainThread) [homeassistant.components.amcrest.camera] async_camera_image start 2019-03-05 17:54:09 CRITICAL (SyncWorker_8) [amcrest.snapshot] snapshot start 2019-03-05 17:54:09 WARNING (SyncWorker_2) [urllib3.connectionpool] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))': /cgi-bin/snapshot.cgi?channel=0 2019-03-05 17:54:09 WARNING (SyncWorker_2) [amcrest.http] Trying again due to error 401 Client Error: Unauthorized for url: http://192.168.1.164:10007/cgi-bin/snapshot.cgi?channel=0

This shows two successful snapshots, but even so, there is a long delay from the snapshot method finishing to async_camera_image finishing. I don't understand that. Then you can see async_camera_image starts twice, which results in two calls to snapshot starting twice, which leads to the errors that the locking is supposed to be avoiding.

Why does the locking seem like it's not working?

Why does the locking seem like it's not working?

Sorry, try this:

async with self._snapshot_lock:

... but please do open a new PR (you can tag me).

Thank you. That fixed it! (And now I see why. Still learning asyncio!)

I'll be submitting a new PR with the change soon.

SukramJ · 2019-03-06T21:41:39Z

Hi,
there is no 1.2.4 in pypi. Just 1.2.3 and 1.2.5.

BR
Markus

pnbruckner · 2019-03-06T21:45:15Z

Yes, I know. It was a misunderstanding between me and the maintainer. Working on a new PR now to change to 1.2.5.

…

On Wed, Mar 6, 2019 at 3:42 PM Markus Jankowski ***@***.***> wrote: Hi, there is no 1.2.4 in pypi. Just 1.2.3 and 1.2.5. BR Markus — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#21664 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AiWM2XAK93IxvzN9h7Q_naPD6d-B1uClks5vUDY1gaJpZM4bdRoo> .

homeassistant added integration: amcrest small-pr PRs with less than 30 lines. cla-signed labels Mar 4, 2019

ghost added the in progress label Mar 4, 2019

Update amcrest package to 1.2.4

08a61b8

amelchio approved these changes Mar 5, 2019

View reviewed changes

amelchio merged commit 16d79b5 into home-assistant:dev Mar 5, 2019

ghost removed the in progress label Mar 5, 2019

pnbruckner deleted the amcrest branch March 5, 2019 16:28

MartinHjelmare reviewed Mar 5, 2019

View reviewed changes

This was referenced Mar 6, 2019

Change amcrest camera_image to async #21720

Merged

Snapshot does not work on Amcrest firmware V2.400.AC02.15.R.20170731 tchellomello/python-amcrest#102

Closed

pnbruckner mentioned this pull request Mar 14, 2019

Bump amcrest to 1.2.6 & use new exceptions #22040

Merged

4 tasks

hunterjm mentioned this pull request Mar 16, 2019

Bump amcrest to 1.2.7 for correct RTSP port #22099

Merged

4 tasks

balloob mentioned this pull request Mar 20, 2019

0.90.0 #22216

Merged

pnbruckner mentioned this pull request Mar 26, 2019

Amcrest Camera Component Retrying ConnectionTimeoutError Repeatedly in Logs #19687

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Serialize amcrest snapshot commands and bump PyPI package to 1.2.4 #21664

Serialize amcrest snapshot commands and bump PyPI package to 1.2.4 #21664

pnbruckner commented Mar 4, 2019 •

edited

Loading

amelchio commented Mar 4, 2019

pnbruckner commented Mar 4, 2019

amelchio commented Mar 5, 2019

pnbruckner commented Mar 5, 2019

amelchio left a comment

amelchio Mar 5, 2019

pnbruckner Mar 5, 2019

MartinHjelmare Mar 5, 2019

amelchio Mar 5, 2019

MartinHjelmare Mar 5, 2019

amelchio Mar 5, 2019

amelchio Mar 5, 2019

pnbruckner Mar 5, 2019

amelchio Mar 5, 2019

pnbruckner Mar 6, 2019

amelchio Mar 6, 2019

pnbruckner Mar 6, 2019

SukramJ commented Mar 6, 2019

pnbruckner commented Mar 6, 2019 via email

Serialize amcrest snapshot commands and bump PyPI package to 1.2.4 #21664

Serialize amcrest snapshot commands and bump PyPI package to 1.2.4 #21664

Conversation

pnbruckner commented Mar 4, 2019 • edited Loading

Description:

Example entry for configuration.yaml (if applicable):

Checklist:

amelchio commented Mar 4, 2019

pnbruckner commented Mar 4, 2019

amelchio commented Mar 5, 2019

pnbruckner commented Mar 5, 2019

amelchio left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SukramJ commented Mar 6, 2019

pnbruckner commented Mar 6, 2019 via email

pnbruckner commented Mar 4, 2019 •

edited

Loading

Example entry for `configuration.yaml` (if applicable):