Problem: debug_traceBlock family should not return Tendermint failed ETH Tx

[calvinaco]

System info: [Include Ethermint commit, operating system name, and other relevant details]

• Ethermint commit: main branch (fcdc625)
• OS: N/A

Steps to reproduce:

1. Fund account A with 100coin
2. Submit 3 ETH transactions, each sending 40coin to any account B, to the mempool
3. First 2 ETH transactions will succeeded, 3rd ETH transaction should failed at ante handler (

   ethermint/app/ante/eth.go

   Line 310 in fcdc625

   if coreMsg.Value().Sign() > 0 && !evm.Context().CanTransfer(stateDB, coreMsg.From(), coreMsg.Value()) {

   )
4. Call debug_traceBlockByNumber or debug_traceBlockByHash on this block
5. The JSON RPC will return a trace object with zero value at the index of the failed transaction

{
    "jsonrpc": "2.0",
    "id": 42,
    "result": [
        {
            "result": {
                "from": "0xebf80ff512d5af394c2f86b39aa92670d6d3b15f",
                "gas": "0x0",
                "gasUsed": "0x0",
                "input": "0x",
                "output": "0x",
                "to": "0xb6a2f15a37aac3b6fdd39afa356b64f51ca8314a",
                "type": "CALL",
                "value": "0x22b1c8c1227a00000"
            }
        },
        {
            "result": {
                "from": "0xebf80ff512d5af394c2f86b39aa92670d6d3b15f",
                "gas": "0x0",
                "gasUsed": "0x0",
                "input": "0x",
                "output": "0x",
                "to": "0xb6a2f15a37aac3b6fdd39afa356b64f51ca8314a",
                "type": "CALL",
                "value": "0x22b1c8c1227a00000"
            }
        },
        {
            "result": {
                "from": "",
                "gas": "",
                "gasUsed": "",
                "input": "",
                "type": ""
            }
        }
    ]
}

Expected behavior: [What you expected to happen]

• ETH transaction failed at Cosmos layer (ante handler) should be ignored

Actual behavior: [What actually happened]

• An zero value trace object is returned at the index of the failed tx

Additional info: [Include gist of relevant config, logs, etc.]

In the process of investigating the solution, I noticed there maybe one more issue that may result in an inevitable behaviour difference on debug_traceBlock* with Ethereum. But further confirmation is needed.

While the failed ETH Tx can be easily filtered in the RPC backend, there's another issue from here. In the current block-related RPCs, it is using the filter TxSuccessOrExceedsBlockGasLimit (https://github.com/evmos/ethermint/blob/main/rpc/backend/blocks.go#L261), that means there are two types of ETH txs returned

1. Tx executed on EVM (regardless of EVM execution result)
2. Tx exceeded block gas limit

For 1/, it is fine and there will be a trace;
For 2/, it is ok to return in most RPC because the ETH Tx can be found from Tendermint RPCs, just need to make sure we return a failed status in calls like eth_getTransactionReceipt. But I think it will not have trace because it is not executed in the EVM. However, I am struggling to reproduce a test case so I am unable to verify the behaviour here.