Dealing with next Hive Hard Fork 26 and testing in advance

[brianoflondon]

We used to have in the code a facility to switch to a testnet API but that was removed.

I'm looking at either putting that back so I can run tests for the upcoming Hard Fork:

https://peakd.com/hive-139531/@blocktrades/14th-update-of-2022-hive-hf26-release-candidate-and-mirrornet-api-node-ready-for-app-testing

The new release candidate has an informal date of Oct 11th for the date of the hardfork. The code requires us to always specify some date, and this was the earliest “feasible” date that we considered for it (feasible in terms of giving sufficient time for other parts of the ecosystem to be prepared for the hardfork). Discussions among various stakeholders supporting the ecosystem (core devs, apps devs, witnesses, api node operators, etc) will be necessary before this date or a later date can be made official. And, of course, it is always possible that we encounter some problematic error despite our quite rigorous testing that could cause the date to be pushed (but I think it is unlikely).

[brianoflondon]

@agates

I had to do a fair bit of work and testing to make RC Depletion handle as a retry. I believe I have all the automated tests working though I will admit they sometimes fail on Main net even though I know they work more reliably on the post HF26 FakeNet (this is desired anyway).

In the end very small code changes.

It's quite important that @daveajones starts running this today I think.

I know we've cut this really tight.

A key point is that I've hard coded the 3 most reliable API servers in. I think this is necessary for the next week or so AT LEAST as I know these three should be reliable. After everything shakes out we might go back to the default list in Lighthive.

        else:
            nodes = [
                "https://hive-api.arcange.eu",
                "https://api.deathwing.me",
                "https://api.hive.blog",
            ]

This is from the automated test and I'll explain below:

2022-10-11 08:34:25 [   ERROR] Failed to send 7 IRIs (podping_hivewriter.py:489)
2022-10-11 08:34:25 [   ERROR]  (podping_hivewriter.py:492)
2022-10-11 08:34:25 [ WARNING] Malformed error response (podping_hivewriter.py:506)
2022-10-11 08:34:25 [    INFO] FAILURE COUNT: 47 - RETRYING 7 IRIs (podping_hivewriter.py:473)
2022-10-11 08:34:27 [   ERROR] send_notification error: payer has not enough RC mana for transaction:Account: *** has 271970563 RC, needs 979072257 RC. Please wait to transact, or power up HIVE. (podping_hivewriter.py:404)
2022-10-11 08:34:27 [   ERROR] payer has not enough RC mana for transaction:Account: *** has 271970563 RC, needs 979072257 RC. Please wait to transact, or power up HIVE. (podping_hivewriter.py:415)
2022-10-11 08:34:27 [   ERROR] Failed to send 7 IRIs (podping_hivewriter.py:489)
2022-10-11 08:34:27 [   ERROR]  (podping_hivewriter.py:492)
2022-10-11 08:34:27 [ WARNING] Malformed error response (podping_hivewriter.py:506)
2022-10-11 08:34:27 [    INFO] FAILURE COUNT: 48 - RETRYING 7 IRIs (podping_hivewriter.py:473)
2022-10-11 08:34:29 [   ERROR] send_notification error: payer has not enough RC mana for transaction:Account: *** has 271970563 RC, needs 979072257 RC. Please wait to transact, or power up HIVE. (podping_hivewriter.py:404)
2022-10-11 08:34:29 [   ERROR] payer has not enough RC mana for transaction:Account: *** has 271970563 RC, needs 979072257 RC. Please wait to transact, or power up HIVE. (podping_hivewriter.py:415)
2022-10-11 08:34:29 [   ERROR] Failed to send 7 IRIs (podping_hivewriter.py:489)
2022-10-11 08:34:29 [   ERROR]  (podping_hivewriter.py:492)
2022-10-11 08:34:29 [ WARNING] Malformed error response (podping_hivewriter.py:506)
2022-10-11 08:34:29 [    INFO] FAILURE COUNT: 49 - RETRYING 7 IRIs (podping_hivewriter.py:473)
2022-10-11 08:34:30 [    INFO] Lighthive Node: https://api.hive.blog/ (podping_hivewriter.py:401)
2022-10-11 08:34:30 [    INFO] JSON size: 682 (podping_hivewriter.py:402)
2022-10-11 08:34:30 [    INFO] FAILURE CLEARED after 49 retries (podping_hivewriter.py:486)
2022-10-11 08:34:30 [    INFO] Batch send time: 101.56 | Failures: 49 - IRI batch_id 4636b5e0-349d-4b70-9a85-dde7ad26bbb8 | IRIs in batch: 7 | last_node: https://api.hive.blog/ (podping_hivewriter.py:236)
PASSED

During the git test I actually managed to run out of Resource Credits for the podping.gittest account. This happened TODAY because the RC Delegation bot I'm running doesn't yet work on Main Net so while this test was stalled, I manually gave more Hive Power to the podping.gittest account. That did the trick and after 49 retries the test continued successfully.

Post HardFork 26 this RC Delegation (instead of a Hive Power delegation) would have happened automatically.

I actually want to change the messages here but right now I'm sure it is working and I'm going to leave it alone. Testing against the HF25 and HF26 chains simultaneously is really hard so I'd rather cosmetically improve this after everything shakes out.

As far as writing test scripts to simulate this I think it is not necessary right now but perhaps I will think about how to do that.