Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

51 non standard rpc error response from hive rpc node #52

Merged
merged 8 commits into from
Aug 6, 2022
Merged

51 non standard rpc error response from hive rpc node #52

merged 8 commits into from
Aug 6, 2022

Conversation

brianoflondon
Copy link
Collaborator

@agates I monitor my logs and I think something I fixed before... needed more fixing.

agates and others added 7 commits June 19, 2022 10:45
@agates
Remove deprecated use of random.sample on sets
0f9759a
@brianoflondon
Dealing with bad error response from Hive RPC
c6d11e1 
Adding some logic to stop this response from Hive bringing down the hivewriter.

```
  File "/home/podping/app/.venv/lib/pypy3.8/site-packages/lighthive/client.py", line 167, in validate_response
    raw_body=response,
lighthive.exceptions.RPCNodeException: Internal Error

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/podping/app/src/podping_hivewriter/podping_hivewriter.py", line 495, in failure_retry
    ex.raw_body["error"]["data"]["name"]
KeyError: 'name'
```
@brianoflondon
Revert "Dealing with bad error response from Hive RPC"
0143b77 
This reverts commit c6d11e1.
@brianoflondon
Fixing issue #51
7a0280e 
dealing with a non standard response from a Hive RPC node
@brianoflondon
add hotfix to version number
ef74534 
Signed-off-by: Brian of London <[email protected]>
@brianoflondon
change version to 1.2.5-alpha.0
0352b71 
Signed-off-by: Brian of London <[email protected]>
@brianoflondon
poetry update dependencies
b317f1c
@brianoflondon brianoflondon linked an issue Aug 6, 2022 that may be closed by this pull request
Non standard RPC error response from Hive RPC node #51
Closed
@brianoflondon brianoflondon changed the base branch from main to develop August 6, 2022 10:26
@agates
Copy link
Collaborator

agates commented Aug 6, 2022

This PR doesn't make any code changes, what am I missing?

@agates
Copy link
Collaborator

agates commented Aug 6, 2022

Oh. You already pushed the code change to develop. Please don't do that.

@agates agates changed the base branch from develop to hotfix/1.2.5 August 6, 2022 19:41
@agates agates merged commit 589c4ac into hotfix/1.2.5 Aug 6, 2022
@brianoflondon
Copy link
Collaborator Author

Oh. You already pushed the code change to develop. Please don't do that.

I kinda messed up and tried to reverse that commit and then put my changes in a branch.

@brianoflondon
Copy link
Collaborator Author

Success... this is the only error I've had for days and the code dealt with it properly:

2022-08-13T13:02:55+0000 | INFO | Lighthive Fastest: https://rpc.ecency.com
2022-08-13T13:02:55+0000 | INFO | Status - Uptime: 7:02:49 | IRIs Received: 46 | IRIs Deduped: 46 | IRIs Sent: 46 | last_node: https://rpc.ecency.com
2022-08-13T13:04:59+0000 | INFO | IRI batch_id 2f810e23-3cd3-43ee-8d20-0254a320f525 - Size of IRIs: 36
2022-08-13T13:04:59+0000 | INFO | Received 1 IRIs
2022-08-13T13:05:03+0000 | INFO | Backing off _send_request(...) for 0.9s (requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='rpc.ecency.com', port=443): Read timed out. (read timeout=3))
2022-08-13T13:05:40+0000 | INFO | Backing off _send_request(...) for 0.1s (requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='rpc.ecency.com', port=443): Read timed out. (read timeout=3))
2022-08-13T13:05:41+0000 | ERROR | send_notification error: transaction expiration exception:
2022-08-13T13:05:41+0000 | ERROR | Failed to send 1 IRIs
Traceback (most recent call last):
  File "/home/podping/app/src/podping_hivewriter/cli/podping.py", line 245, in server
    loop = asyncio.get_running_loop()
RuntimeError: no running event loop

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/podping/app/src/podping_hivewriter/podping_hivewriter.py", line 482, in failure_retry
    await self.send_notification_iris(
  File "/home/podping/app/src/podping_hivewriter/podping_hivewriter.py", line 456, in send_notification_iris
    await self.send_notification(payload.dict(), hive_operation_id)
  File "/home/podping/app/src/podping_hivewriter/podping_hivewriter.py", line 416, in send_notification
    raise ex
  File "/home/podping/app/src/podping_hivewriter/podping_hivewriter.py", line 401, in send_notification
    await self._async_hive_broadcast(op=op, dry_run=self.dry_run)
  File "/home/podping/app/src/podping_hivewriter/async_wrapper.py", line 32, in wrapper
    return await async_fn(*args, **kwargs)
  File "/home/podping/app/.venv/lib/python3.10/site-packages/asgiref/sync.py", line 435, in __call__
    ret = await asyncio.wait_for(future, timeout=None)
  File "/usr/local/lib/python3.10/asyncio/tasks.py", line 408, in wait_for
    return await fut
  File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/podping/app/.venv/lib/python3.10/site-packages/asgiref/sync.py", line 474, in thread_handler
    return func(*args, **kwargs)
  File "/home/podping/app/.venv/lib/python3.10/site-packages/lighthive/client.py", line 184, in broadcast
    return self.transaction_builder.broadcast(
  File "/home/podping/app/.venv/lib/python3.10/site-packages/lighthive/broadcast/transaction_builder.py", line 202, in broadcast
    resp = self.client('condenser_api').broadcast_transaction(
  File "/home/podping/app/.venv/lib/python3.10/site-packages/lighthive/client.py", line 53, in callable
    return self.request(attr, *args, **kwargs)
  File "/home/podping/app/.venv/lib/python3.10/site-packages/lighthive/client.py", line 152, in request
    self.validate_response(response)
  File "/home/podping/app/.venv/lib/python3.10/site-packages/lighthive/client.py", line 164, in validate_response
    raise RPCNodeException(
lighthive.exceptions.RPCNodeException: transaction expiration exception:
2022-08-13T13:05:41+0000 | ERROR | transaction expiration exception:
Traceback (most recent call last):
  File "/home/podping/app/src/podping_hivewriter/cli/podping.py", line 245, in server
    loop = asyncio.get_running_loop()
RuntimeError: no running event loop

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/podping/app/src/podping_hivewriter/podping_hivewriter.py", line 482, in failure_retry
    await self.send_notification_iris(
  File "/home/podping/app/src/podping_hivewriter/podping_hivewriter.py", line 456, in send_notification_iris
    await self.send_notification(payload.dict(), hive_operation_id)
  File "/home/podping/app/src/podping_hivewriter/podping_hivewriter.py", line 416, in send_notification
    raise ex
  File "/home/podping/app/src/podping_hivewriter/podping_hivewriter.py", line 401, in send_notification
    await self._async_hive_broadcast(op=op, dry_run=self.dry_run)
  File "/home/podping/app/src/podping_hivewriter/async_wrapper.py", line 32, in wrapper
    return await async_fn(*args, **kwargs)
  File "/home/podping/app/.venv/lib/python3.10/site-packages/asgiref/sync.py", line 435, in __call__
    ret = await asyncio.wait_for(future, timeout=None)
  File "/usr/local/lib/python3.10/asyncio/tasks.py", line 408, in wait_for
    return await fut
  File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/podping/app/.venv/lib/python3.10/site-packages/asgiref/sync.py", line 474, in thread_handler
    return func(*args, **kwargs)
  File "/home/podping/app/.venv/lib/python3.10/site-packages/lighthive/client.py", line 184, in broadcast
    return self.transaction_builder.broadcast(
  File "/home/podping/app/.venv/lib/python3.10/site-packages/lighthive/broadcast/transaction_builder.py", line 202, in broadcast
    resp = self.client('condenser_api').broadcast_transaction(
  File "/home/podping/app/.venv/lib/python3.10/site-packages/lighthive/client.py", line 53, in callable
    return self.request(attr, *args, **kwargs)
  File "/home/podping/app/.venv/lib/python3.10/site-packages/lighthive/client.py", line 152, in request
    self.validate_response(response)
  File "/home/podping/app/.venv/lib/python3.10/site-packages/lighthive/client.py", line 164, in validate_response
    raise RPCNodeException(
lighthive.exceptions.RPCNodeException: transaction expiration exception:
2022-08-13T13:05:41+0000 | WARNING | Waiting 3s before retry
2022-08-13T13:05:44+0000 | INFO | FAILURE COUNT: 1 - RETRYING 1 IRIs
2022-08-13T13:05:55+0000 | INFO | Lighthive Node: https://rpc.ausbit.dev
2022-08-13T13:05:55+0000 | INFO | JSON size: 96
2022-08-13T13:05:55+0000 | INFO | FAILURE CLEARED after 1 retries, 3s

@daveajones
Copy link
Contributor

If the exception is being caught and handled gracefully, is there any way to clean that up and make it less of a huge log barf?

@agates
Copy link
Collaborator

agates commented Aug 15, 2022

If the exception is being caught and handled gracefully, is there any way to clean that up and make it less of a huge log barf?

I think we can stop logging known exception tracebacks now that we know what's working.

@agates agates deleted the 51-non-standard-rpc-error-response-from-hive-rpc-node branch August 15, 2022 17:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@agates agates agates approved these changes

Assignees

@agates agates

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Non standard RPC error response from Hive RPC node
3 participants
@brianoflondon @agates @daveajones