51 non standard rpc error response from hive rpc node (#52)

* Remove deprecated use of random.sample on sets

* Dealing with bad error response from Hive RPC

Adding some logic to stop this response from Hive bringing down the hivewriter.

```
  File "/home/podping/app/.venv/lib/pypy3.8/site-packages/lighthive/client.py", line 167, in validate_response
    raw_body=response,
lighthive.exceptions.RPCNodeException: Internal Error

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/podping/app/src/podping_hivewriter/podping_hivewriter.py", line 495, in failure_retry
    ex.raw_body["error"]["data"]["name"]
KeyError: 'name'
```

* Revert "Dealing with bad error response from Hive RPC"

This reverts commit c6d11e1.

* Fixing issue #51

dealing with a non standard response from a Hive RPC node

* add hotfix to version number

Signed-off-by: Brian of London <[email protected]>

* change version to 1.2.5-alpha.0

Signed-off-by: Brian of London <[email protected]>

* poetry update dependencies

Co-authored-by: Alecks Gates <[email protected]>

pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "podping-hivewriter"
-version = "1.2.4"
+version = "1.2.5-alpha.0"
 license = "MIT"
 authors = ["Alecks Gates <[email protected]>", "Brian of London <[email protected]>"]
 maintainers = ["Alecks Gates <[email protected]>", "Brian of London <[email protected]>"]

setup.py

@@ -29,7 +29,7 @@
 
 setup_kwargs = {
     "name": "podping-hivewriter",
-    "version": "1.2.4",
+    "version": "1.2.5-alpha.0",
     "description": "This is a tool used to submit RFC 3987-compliant International Resource Identifiers as a Podping notification on the Hive blockchain.",
     "long_description": "# podping-hivewriter\nThe Hive writer component of Podping. You will need a Hive account, see section [Hive account and Authorization](#hive-account) below.\n\n## What is Podping?\n\nPodping is a mechanism of using decentralized communication to relay notification of updates of RSS feeds that use The Podcast Namespace.  It does so by supplying minimum relevant metadata to consumers to be able to make efficient and actionable decisions, allowing them to decide what to do with given RSS feeds without parsing them ahead of time.\n\n*This* project provides a standardized way of posting a \"podping\" specifically to the Hive blockcahin.\n\n## Running podping-hivewriter\n\nThe project has two modes of running:  `write` mode and `server` mode.\n\n`write` mode is primarily useful for people with a very small number of feeds to publish updates for relatively infrequently (i.e. a few times a day or less).\n\n`server` mode is for hosts (or other services like the Podcast Index's [podping.cloud](https://podping.cloud/)) who publish updates for a significant amount of feeds on a regular basis.  Not that the average small-time podcast can't run it, but it's overkill.  This mode is for efficiency only, as the `server` will batch process feeds as they come in to make the most use of the Hive blockchain.\n\nSee the dedicated [CLI docs](CLI.md) for more information on configuration options, including environment variables.\n\n### Container\n\nThe container images are hosted on [Docker Hub](https://hub.docker.com/r/podcastindexorg/podping-hivewriter).  Images are currently based on Debian bullseye-based PyPy 3.8 with the following architectures: `amd64`\n\nThese images can be run in either `write` or `server` mode and is likely the easiest option for users who do not have experience installing Python packages.\n\n#### Command Line\n\nRunning in `write` mode with command line options, like `--dry-run` for example, add them with the full podping command.\nSettings can also be passed with the `-e` option for Docker.  Note, we leave out `-p 9999:9999` here because we're not running the server.\n\n```shell\ndocker run --rm \\\n    -e PODPING_HIVE_ACCOUNT=<account> \\\n    -e PODPING_HIVE_POSTING_KEY=<posting-key> \\\n    docker.io/podcastindexorg/podping-hivewriter \\\n    --dry-run write https://www.example.com/feed.xml\n```\n\nRun in `server` mode, passing local port 9999 to port 9999 in the container.\nENV variables can be passed to docker with `--env-file` option after modifying the `.env.EXAMPLE` file and renaming it to `.env`\n\n```shell\ndocker run --rm -p 9999:9999 --env-file .env --name podping docker.io/podcastindexorg/podping-hivewriter\n```\n\nAs another example for running in `server` mode, to run in *detached* mode, note the `-d` in the `docker run` options.  Also note that `write` or `server` must come *after* the command line options for `podping`:\n```shell\ndocker run --rm -d \\\n    -p 9999:9999 --env-file .env \\\n    --name podping \\\n    docker.io/podcastindexorg/podping-hivewriter \\\n    --livetest server\n```\n\nOne running you can view and follow the live output with:\n```shell\ndocker logs podping -f\n```\n\nSee the [CLI docs](https://github.com/Podcastindex-org/podping-hivewriter/blob/main/CLI.md) for default values.\n\n\n#### docker-compose\n\n```yaml\nversion: '2.0'\nservices:\n  podping-hivewriter:\n    image: docker.io/podcastindexorg/podping-hivewriter\n    restart: always\n    ports:\n      - \"9999:9999\"\n    environment:\n      - PODPING_HIVE_ACCOUNT=<account>\n      - PODPING_HIVE_POSTING_KEY=<posting-key>\n      - PODPING_LISTEN_IP=0.0.0.0\n      - PODPING_LISTEN_PORT=9999\n      - PODPING_LIVETEST=false\n      - PODPING_DRY_RUN=false\n      - PODPING_STATUS=true\n      - PODPING_IGNORE_CONFIG_UPDATES=false\n      - PODPING_I_KNOW_WHAT_IM_DOING=false\n      - PODPING_DEBUG=false\n```\n\nAssuming you just copy-pasted without reading, the above will fail at first.  As noted in the [server command documentation](https://github.com/Podcastindex-org/podping-hivewriter/blob/main/CLI.md#podping-server):\n\n>WARNING: DO NOT run this on a publicly accessible host. There currently is NO authentication required to submit to the server. Set to * or 0.0.0.0 for all interfaces.\n\nAs all Docker installations vary, we set `0.0.0.0` as the listen IP for connectivity.  This doesn't affect the IP address docker listens on when we tell it to pass port `9999` through to the container.  If you understand the consequences of this, set `PODPING_I_KNOW_WHAT_IM_DOING` to `true`.\n\nThis is a temporary measure to limit potential misconfiguration until we fully bundle the `podping.cloud` HTTP front end.  Then again, if you're running this, you're probably Dave.\n\n\n### CLI Install\n\nThe following have been tested on Linux and macOS.  However, Windows should work also.  If you have issues on Windows we highly recommend the [Windows Subsystem for Linux](https://docs.microsoft.com/en-us/windows/wsl/) and/or Docker.\n\n#### Using [pipx](https://pypa.github.io/pipx/) (preferred over pip)\n```shell\npipx install podping-hivewriter\n```\n\n#### Using pip\n```shell\npip install --user podping-hivewriter\n```\n\n#### Installing the server\n\nIf you'd like to install the server component, it's hidden behind the extra flag `server`.  This is to make it easier to install only the `write` CLI component `podping-hivewriter` on non-standard systems without a configured development enviornment.\n\n```shell\npipx install podping-hivewriter[server]\n```\n\nMake sure you have `~/.local/bin/` on your `PATH`.\n\nSee the dedicated [CLI docs](CLI.md) for more information.\n\n## Podping reasons\n\nPodping accepts various different \"reasons\" for publishing updates to RSS feeds:\n\n* `update` -- A general indication that an RSS feed has been updated\n* `live` -- An indication that an RSS feed has been updated and a contained [`<podcast:liveItem>`](https://github.com/Podcastindex-org/podcast-namespace/blob/main/docs/1.0.md#live-item) tag's status attribute has been changed to live.\n* `liveEnd` -- An indication that an RSS feed has been updated and either the status attribute of an existing [`<podcast:liveItem>`](https://github.com/Podcastindex-org/podcast-namespace/blob/main/docs/1.0.md#live-item) has been changed from live to ended or a [`<podcast:liveItem>`](https://github.com/Podcastindex-org/podcast-namespace/blob/main/docs/1.0.md#live-item) that previously had a status attribute of live has been removed from the feed entirely.\n\nThe canonical list of reasons within the scope of this project is [maintained in this schema](https://github.com/Podcastindex-org/podping-hivewriter/blob/main/src/podping_hivewriter/schema/reason.capnp).\n\n## Mediums\n\nPodping accepts various different \"mediums\" for identifying types of RSS feeds using the Podcast Namespace.  Please check the [`<podcast:medium>`](https://github.com/Podcastindex-org/podcast-namespace/blob/main/docs/1.0.md#medium) specification for the full list.\n\n`podping-hivewriter` *may* lag behind the specification, and if it does, please let us know or submit a pull request.\n\nThe canonical list of mediums within the scope of this project is [maintained in this schema](https://github.com/Podcastindex-org/podping-hivewriter/blob/main/src/podping_hivewriter/schema/medium.capnp).\n\n## Development\n\nYou'll need a few extras:\n\n1. [capnproto](https://capnproto.org/). Linux: `capnproto` package in your package manager.  On a Mac: `brew instal capnp`\n2. [Poetry](https://python-poetry.org/docs/)\n\n\nWe use [poetry](https://python-poetry.org/) for dependency management.  Once you have it, clone this repo and run:\n\n```shell\npoetry install\n```\n\nThen to switch to the virtual environment, use:\n\n```shell\npoetry shell\n```\nMake sure you have a the environment variables `PODPING_HIVE_ACCOUNT` and `PODPING_HIVE_POSTING_KEY` set.\n\nAfter that you should be able to run the `podping` command or run the tests:\n\n```shell\npytest\n```\n\nTo run all tests, make sure to set the necessary environment variables for your Hive account.  This will take many minutes:\n\n```shell\npytest --runslow\n```\n\n### Building the image locally with Docker\n\nLocally build the podping-hivewriter container with a \"develop\" tag\n\n```shell\ndocker build -t podping-hivewriter:develop .\n```\n\nSee above for more details on running the docker CLI.\n\n## Hive account\n\nIf you need a Hive account, please download the [Hive Keychain extension for your browser](https://hive-keychain.com/) then use this link to get your account from [https://HiveOnboard.com?ref=podping](https://hiveonboard.com?ref=podping). You will need at least 20 Hive Power \"powered up\" to get started (worth around $10). Please contact [@brianoflondon](https://peakd.com/@brianoflondon) [email protected] if you need assistance getting set up.\n\nIf you use the [Hiveonboard]((https://hiveonboard.com?ref=podping)) link `podping` will **delegate** enough Hive Power to get you started. If, for any reason, Hiveonboard is not giving out free accounts, please contact [@brianoflondon](https://peakd.com/@brianoflondon) either on [PodcastIndex Social](https://podcastindex.social/invite/U2m6FY3T) or [Telegram](https://t.me/brianoflondon).\n\n### Permissions and Authorization\n\nYou don't need permission, but you do need to tell `podping` that you want to send valid `podpings`:\n\n- Hive is a so-called \"permissionless\" blockchain.  Once you have a Hive Account and a minimal amount of Hive Power, that account can post to Hive, including sending `podpings`.\n- Nobody can block any valid Hive Account from sending and nobody can help you if you lose your keys.\n- Whilst anyone can post `podpings` to Hive, there is a need to register your Hive Accountname for those `podpings` to be recognized by all clients.  This is merely a spam-prevention measure and clients may choose to ignore it.\n- Please contact [email protected] or send a Hive Transfer to [@podping](https://peakd.com/@podping) to have your account validated.\n- Side note on keys: `podping` uses the `posting-key` which is the lowest value of the four Hive keys (`owner`, `active`, `memo`, `posting` and there is usually a `master password` which can generate all the keys). That is not to say that losing control of it is a good idea, but that key is not authorized to make financially important transfers. It can, however, post public information so should be treated carefully and kept secure.\n\nFor a [comprehensive explanation of Hive and Podping, please see this post](https://peakd.com/podping/@brianoflondon/podping-and-podcasting-20-funding-to-put-hive-at-the-center-of-global-podcasting-infrastructure).",
     "author": "Alecks Gates",

src/podping_hivewriter/podping_hivewriter.py

@@ -491,25 +491,33 @@ async def failure_retry(
             except RPCNodeException as ex:
                 logging.exception(f"Failed to send {len(iri_set)} IRIs")
                 try:
+                    # Test if we have a well formed Hive error message
+                    logging.exception(ex)
                     if (
-                        ex.raw_body["error"]["data"]["name"]
-                        == "tx_missing_posting_auth"
+                        ex.raw_body.get("error")
+                        and ex.raw_body["error"].get("data")
+                        and ex.raw_body["error"]["data"].get("name")
                     ):
-                        if logging.DEBUG >= logging.root.level:
-                            for iri in iri_set:
-                                logging.debug(iri)
-                        logging.error(
-                            f"Terminating: exit code: "
-                            f"{STARTUP_FAILED_INVALID_POSTING_KEY_EXIT_CODE}"
-                        )
-                        sys.exit(STARTUP_FAILED_INVALID_POSTING_KEY_EXIT_CODE)
+                        if (
+                            ex.raw_body["error"]["data"]["name"]
+                            == "tx_missing_posting_auth"
+                        ):
+                            if logging.DEBUG >= logging.root.level:
+                                for iri in iri_set:
+                                    logging.debug(iri)
+                            logging.error(
+                                f"Terminating: exit code: "
+                                f"{STARTUP_FAILED_INVALID_POSTING_KEY_EXIT_CODE}"
+                            )
+                            sys.exit(STARTUP_FAILED_INVALID_POSTING_KEY_EXIT_CODE)
                 except Exception:
                     logging.info(f"Current node: {self.lighthive_client.current_node}")
                     logging.info(self.lighthive_client.nodes)
                     logging.exception("Unexpected condition in error text from Hive")
                     sys.exit(STARTUP_FAILED_UNKNOWN_EXIT_CODE)
 
             except Exception as ex:
+                logging.exception(ex)
                 logging.exception(f"Failed to send {len(iri_set)} IRIs")
                 if logging.DEBUG >= logging.root.level:
                     for iri in iri_set:

tests/integration/test_write_cli_multiple.py

@@ -35,8 +35,8 @@ async def test_write_cli_multiple():
         for i in range(num_iris)
     }
 
-    medium = str_medium_map[random.sample(mediums, 1)[0]]
-    reason = str_reason_map[random.sample(reasons, 1)[0]]
+    medium = str_medium_map[random.sample(sorted(mediums), 1)[0]]
+    reason = str_reason_map[random.sample(sorted(reasons), 1)[0]]
 
     default_hive_operation_id = HiveOperationId(LIVETEST_OPERATION_ID, medium, reason)
     default_hive_operation_id_str = str(default_hive_operation_id)

tests/integration/test_write_cli_single.py

@@ -28,8 +28,8 @@ async def test_write_cli_single():
     test_name = "cli_single"
     iri = f"https://example.com?t={test_name}&v={pv()}&s={session_uuid_str}"
 
-    medium = str_medium_map[random.sample(mediums, 1)[0]]
-    reason = str_reason_map[random.sample(reasons, 1)[0]]
+    medium = str_medium_map[random.sample(sorted(mediums), 1)[0]]
+    reason = str_reason_map[random.sample(sorted(reasons), 1)[0]]
 
     default_hive_operation_id = HiveOperationId(LIVETEST_OPERATION_ID, medium, reason)
     default_hive_operation_id_str = str(default_hive_operation_id)

tests/integration/test_write_cli_single_simulcast.py

@@ -53,8 +53,8 @@ async def get_iri_from_blockchain(start_block: int):
         for n in range(7)
     }
     for iri in test_iris:
-        medium = str_medium_map[random.sample(mediums, 1)[0]]
-        reason = str_reason_map[random.sample(reasons, 1)[0]]
+        medium = str_medium_map[random.sample(sorted(mediums), 1)[0]]
+        reason = str_reason_map[random.sample(sorted(reasons), 1)[0]]
         args = [
             "--medium",
             str(medium),

tests/integration/test_write_zmq_multiple.py

@@ -39,8 +39,8 @@ async def test_write_zmq_multiple(event_loop):
         for i in range(num_iris)
     }
 
-    medium = str_medium_map[random.sample(mediums, 1)[0]]
-    reason = str_reason_map[random.sample(reasons, 1)[0]]
+    medium = str_medium_map[random.sample(sorted(mediums), 1)[0]]
+    reason = str_reason_map[random.sample(sorted(reasons), 1)[0]]
 
     default_hive_operation_id = HiveOperationId(LIVETEST_OPERATION_ID, medium, reason)
     default_hive_operation_id_str = str(default_hive_operation_id)

tests/integration/test_write_zmq_single.py

@@ -32,8 +32,8 @@ async def test_write_zmq_single(event_loop):
     test_name = "zmq_single"
     iri = f"https://example.com?t={test_name}&v={pv()}&s={session_uuid_str}"
 
-    medium = str_medium_map[random.sample(mediums, 1)[0]]
-    reason = str_reason_map[random.sample(reasons, 1)[0]]
+    medium = str_medium_map[random.sample(sorted(mediums), 1)[0]]
+    reason = str_reason_map[random.sample(sorted(reasons), 1)[0]]
 
     default_hive_operation_id = HiveOperationId(LIVETEST_OPERATION_ID, medium, reason)
     default_hive_operation_id_str = str(default_hive_operation_id)

tests/regression/test_#48_failure_retry_handles_invalid_error_response.py

@@ -33,8 +33,8 @@ def mock_broadcast(*args, **kwargs):
     test_name = "failure_retry_handles_invalid_error_response"
     iri = f"https://example.com?t={test_name}&v={pv()}&s={session_uuid_str}"
 
-    medium = str_medium_map[random.sample(mediums, 1)[0]]
-    reason = str_reason_map[random.sample(reasons, 1)[0]]
+    medium = str_medium_map[random.sample(sorted(mediums), 1)[0]]
+    reason = str_reason_map[random.sample(sorted(reasons), 1)[0]]
 
     host = "127.0.0.1"
     port = 9979