Missing transaction from ledger 6409247

[alecalve]

Issue Description

Ledger 6409247 contains only a single (failed) transaction according to XRPScan and s2.ripple.com.

However, ledger 6409248 contains a reference to a transaction which is supposed to be in ledger 6409247.

Expected Result

I would have expected ledger 6409247 to contain transaction DB53018AAFBD1E4AE9C69C82AAB6535E9A9A969F1133E4F79DAE58319C9CFD63

[ihomp]

Shouldn't 6409247 have at least 2 more transactions then? as the tx included there has an index "2", where is the tx with index 0 and index 1?

[alecalve]

That's a good question, I tried to find whether the index had to start at 0 or 1 in rippled but couldn't find it

[RichardAH]

These are very old ledgers. Looks like a case of missing history
Reply from s2 is below. Notice that the transaction_hash is not 0 but the number of transactions is zero. This indicates the server has the ledger header but not the transactions for that ledger.

    "result": {
        "ledger": {
            "accepted": true,
            "account_hash": "F9824325C14F1C34A8C5AF283A887A8DCBFE218AEEFCCA843281596B1C350C01",
            "close_flags": 0,
            "close_time": 452387070,
            "close_time_human": "2014-May-02 23:04:30.000000000 UTC",
            "close_time_resolution": 10,
            "closed": true,
            "hash": "E14C90C59CA0198D50CC2A863BD183680F8EFE3C75DA50BBD39F9D89A46B76A2",
            "ledger_hash": "E14C90C59CA0198D50CC2A863BD183680F8EFE3C75DA50BBD39F9D89A46B76A2",
            "ledger_index": "6409247",
            "parent_close_time": 452387060,
            "parent_hash": "49C7DF14A2F07018A032D757838C19DAB992D8A0D044BDFFE57E938B8AE0AB50",
            "seqNum": "6409247",
            "totalCoins": "99999993536372126",
            "total_coins": "99999993536372126",
            "transaction_hash": "99B6EDE9F3C06B1AC63C9E71015BC02AC138DFC7756B3C5B8F8C2AA83E0C03FC",
            "transactions": []
        },
        "ledger_hash": "E14C90C59CA0198D50CC2A863BD183680F8EFE3C75DA50BBD39F9D89A46B76A2",
        "ledger_index": 6409247,
        "validated": true,

[alecalve]

How would it be possible to detect/obtain these missing transactions, ideally in an automated fashion?

[alecalve]

For example it looks like the other tx missing from 6409247 is ABF77E680C8E88225D74FF54F06993C7B3A3D49A896BE62D017E4F4D05832AE9 and it's not present in s2

[alecalve]

As for the number of transactions returned by s2 for 6409247, I am getting a different result using the old API:

$ curl -s "https://data.ripple.com/v2/ledgers/6409247?transactions=true" | jq
{
  "result": "success",
  "ledger": {
    "account_hash": "F9824325C14F1C34A8C5AF283A887A8DCBFE218AEEFCCA843281596B1C350C01",
    "close_flags": "0",
    "close_time": 1399071870,
    "close_time_human": "2014-May-02 23:04:30",
    "close_time_resolution": "10",
    "hash": "E14C90C59CA0198D50CC2A863BD183680F8EFE3C75DA50BBD39F9D89A46B76A2",
    "ledger_hash": "E14C90C59CA0198D50CC2A863BD183680F8EFE3C75DA50BBD39F9D89A46B76A2",
    "ledger_index": 6409247,
    "parent_close_time": 1399071860,
    "parent_hash": "49C7DF14A2F07018A032D757838C19DAB992D8A0D044BDFFE57E938B8AE0AB50",
    "seqNum": "6409247",
    "totalCoins": "99999993536372126",
    "total_coins": "99999993536372126",
    "transaction_hash": "99B6EDE9F3C06B1AC63C9E71015BC02AC138DFC7756B3C5B8F8C2AA83E0C03FC",
    "transactions": [
      "93AD2A563879DF73F1EADEF6ABDF1D268B4C9ABC30536395CF9F9435055329AC"
    ],
    "tx_count": "1"
  }
}

[cjcobb23]

data.ripple.com is not s2. s2 is s2.ripple.com. data.ripple.com is a totally different server

[alecalve]

I meant that s2 says there's 0 transactions, data.ripple.com 1 but it seems there's at least 3 since there's a transaction with index=2

[cjcobb23]

There are transactions that rippled can no longer deserialize. I am not sure if that's the issue here, but it could be related. See here: #4323

[alecalve]

ah that would explain why s2 returns an internal error when querying ABF77E680C8E88225D74FF54F06993C7B3A3D49A896BE62D017E4F4D05832AE8 instead of returning txnNotFound if it was really unknown:

$ curl -s 'http://s2.ripple.com:51234' -X POST -H "Content-Type: application/json" -d '{"jsonrpc": "2.0","id":1,"method":"tx","params":[{"transaction":"ABF77E680C8E88225D74FF54F06993C7B3A3D49A896BE62D017E4F4D05832AE9"}]}' | jq ".result"
{
  "error": "internal",
  "error_code": 73,
  "error_message": "Internal error.",
  "request": {
    "command": "tx",
    "transaction": "ABF77E680C8E88225D74FF54F06993C7B3A3D49A896BE62D017E4F4D05832AE9"
  },
  "status": "error",
  "warnings": [
    {
      "id": 1004,
      "message": "This is a reporting server.  The default behavior of a reporting server is to only return validated data. If you are looking for not yet validated data, include \"ledger_index : current\" in your request, which will cause this server to forward the request to a p2p node. If the forward is successful the response will include \"forwarded\" : \"true\""
    }
  ]
}

[ihomp]

That's a good question, I tried to find whether the index had to start at 0 or 1 in rippled but couldn't find it

It starts with 0, so the transaction with index 0 is missing.

If you query wss://xrplcluster.com, depending on what full history node it will connect you to, you will get different results. Rather all transactions are missing, or just one is shown.

[cjcobb23]

There are several ledgers where this occurs. I unfortunately don't have a list. Also, it should be noted, as soon as rippled encounters one transaction it can't deserialize, the loop terminates and later transactions in the same ledger are not returned as part of the ledger RPC. Here is the code:

rippled/src/ripple/app/ledger/impl/LedgerToJson.cpp

Line 195 in ebbf4b6

catch (std::exception const&)

[Silkjaer]

To my knowledge, these are the only 4 problematic ledgers: 562177, 6409247, 7266393 and 7266396

[MarkusTeufelberger]

Also afaik the data is actually there, it just can be hard to query it from rippled. If you look in the node store, the actual raw data is present.

[intelliot]

This manifests as the completed_ledgers field from server_info changing from time to time — missing ledger 6409247.

i.e. sometimes, it’s "complete_ledgers" : "32570-80308436". Other times, it’s "complete_ledgers" : "32570-6409246,6409248-80308438".

Here is another thread about this issue: https://groups.google.com/g/ripple-server/c/LXH80_aKomQ

The DB query data matches that of the ledger. You can run the query on your instance of the DB in order to fix the problem.