Improve Ethereum Ledger Settlement Engine robustnessss

[gakonst]

The engine implemented in #125 works but can be improved. In particular:

• When asking for a peer's address, send a challenge which they must sign so that they prove ownership of the key. Otherwise, it's pretty trivial to pretend you are someone else and get the benefit of their settlements Settlement Engine Framework and Ethereum Ledger Settlement Engine #125 (comment)
• extend speculative nonces middleware
• reduce callback hell Settlement Engine Framework and Ethereum Ledger Settlement Engine #125 (comment),
• return futures and do join_all (or similar) instead of spawning: Settlement Engine Framework and Ethereum Ledger Settlement Engine #125 (comment), Settlement Engine Framework and Ethereum Ledger Settlement Engine #125 (comment)
• estimate gas and gas price before sending a transaction Settlement Engine Framework and Ethereum Ledger Settlement Engine #125 (comment), allow node operator to specify urgency similar to ethgasstation
• If a transaction is known to have failed, add a a mechanism to persist the amount that account is owed so it could be retried late.
• Save gas fees by querying the sender's balance before signing a transaction and sending it only if the sender has sufficient balance (ethers.js does that automatically when it calls eth_estimateGas under the hood, need to check if web3-rust does that check) Settlement Engine Framework and Ethereum Ledger Settlement Engine #125 (comment)
• add test: on what happens if the SE dies after making some of the POST requests to the connector but before it saves the current block number? It should be expected to start again and will try to resubmit the same blocks, however it will not submit the same transaction twice, since the check_tx_credited assertion is sufficient to ensure that a transaction is not processed again. Settlement Engine Framework and Ethereum Ledger Settlement Engine #125 (comment)
• optimize calls to connector in case multiple incoming transactions are found from the same account: Settlement Engine Framework and Ethereum Ledger Settlement Engine #125 (comment)
• document edge cases: e.g. If the SE goes down for many blocks and then restarts, this will result in a burst of simultaneous requests, which may fail if using Infura or other ETH providers if they're rate limited. Settlement Engine Framework and Ethereum Ledger Settlement Engine #125 (comment)
• Ensure that the if new_balance > last_observed_balance check works as intended even when the operator moves their funds to another address when an incoming payment is received (otherwise remove that if since it might be a redundant optimization) Settlement Engine Framework and Ethereum Ledger Settlement Engine #125 (comment)
• save recently observed block/balance only when all other operations succeed. If any of the tx settlements fails, it will fetch the blocks since the last successful checkpoint and reexecute all the transactions. The check_tx_credited check will ensure that no transaction gets sent twice.
• Manage incoming ERC20 transfers with event filtering, example: https://github.com/tomusdrw/rust-web3/blob/master/examples/simple_log_filter.rs.
• Test that settle_to works with both sending ETH and an ERC20 token amount

ERC20 Filter guide:

1. Setup filter builder with from/to block https://tomusdrw.github.io/rust-web3/web3/types/struct.FilterBuilder.html
2. Add the filter for transfer(address,uint256) where we filter for address == our_address
3. For each hit, notify the connector

[ALLiDoizCode]

Is there a mismatch in the rust implementations? In the es-examples branch, the settlement engine URL is passed as a property
{ "ilp_address": "example.node-a.mike", "asset_code": "ABC", "asset_scale": 9, "max_packet_amount": 1000, "http_incoming_token": "mike", "settlement_engine_url": "http://localhost:8080/rcl" }

But when I build and run this branch and create an account with the above SE URL followed with a post request to the /pay route my SE endpoint is never hit. Also, all of the endpoints mentioned in the spec are being exposed.

[gakonst]

add test: on what happens if the SE dies after making some of the POST requests to the connector but before it saves the current block number? It should be expected to start again and will try to resubmit the same blocks, however it will not submit the same transaction twice, since the check_tx_credited assertion is sufficient to ensure that a transaction is not processed again. #125 (comment)

Don't think we can make the function call crash during execution.

document edge cases: e.g. If the SE goes down for many blocks and then restarts, this will result in a burst of simultaneous requests, which may fail if using Infura or other ETH providers if they're rate limited. #125 (comment)

This was encountered and fixed by Evan in 99604cf

optimize calls to connector in case multiple incoming transactions are found from the same account: #125 (comment)

Rather not implement this at the moment until we actually need it, premature optimization.