Synchronization stopped with Error in block freeze operation

[mone-d]

System information

Geth version: 1.10.23
Prysm version: 3.0.0
OS & Version: Linux
Commit hash : -

Expected behaviour

No errors have occurred.

Actual behaviour

Error occurred and block synchronization stopped.

Steps to reproduce the behaviour

I started geth with new instance. so instance have no old blocks and geth started synchronization from genesis.
Geth finished initial syncing process with snap sync.
Some days later, I found this log and block synchronization is stopped.
ERROR[09-05|02:07:15.340] Error in block freeze operation err="block receipts missing, can't freeze block 15377501"

My geth configuration.

toml

[Eth]
    NetworkId = 1
    SyncMode = "snap"
    NoPruning = false
    LightPeers = 100
    DatabaseCache = 768
    TrieCleanCache = 256
    TrieDirtyCache = 256
    TrieTimeout = 3600000000000
    EnablePreimageRecording = false
[Eth.Miner]
    GasFloor = 8000000
    GasCeil = 8000000
    GasPrice = 1000000000
    Recommit = 3000000000
    Noverify = false

[Eth.Ethash]
    CacheDir = "ethash"
    CachesInMem = 2
    CachesOnDisk = 3
    DatasetsInMem = 1
    DatasetsOnDisk = 2
    PowMode = 0

[Eth.TxPool]
    Locals = []
    NoLocals = false
    Journal = "transactions.rlp"
    Rejournal = 3600000000000
    PriceLimit = 1
    PriceBump = 10
    AccountSlots = 16
    GlobalSlots = 4096
    AccountQueue = 64
    GlobalQueue = 1024
    Lifetime = 10800000000000

[Eth.GPO]
    Blocks = 20
    Percentile = 60

[Node]
    DataDir             = "/home/app/.ethereum"
    IPCPath = "geth.ipc"

    HTTPHost            = "0.0.0.0"
    HTTPPort            = 8545
    HTTPCors            = ["*"]
    HTTPVirtualHosts    = ["*"]
    HTTPModules         = ["admin", "eth", "debug", "miner", "net", "txpool", "personal", "web3"]

    WSHost              = "0.0.0.0"
    WSPort              = 8546
    WSOrigins           = ["*"]
    WSModules           = ["admin", "eth", "debug", "miner", "net", "txpool", "personal", "web3"]

[Node.P2P]
    MaxPeers = 100
    NoDiscovery = false
    ListenAddr = ":30303"
    EnableMsgEvents = false

[Node.HTTPTimeouts]
    ReadTimeout = 30000000000
    WriteTimeout = 30000000000
    IdleTimeout = 120000000000

Start commands.

geth --config /home/app/.ethereum/config/config.toml --cache 1024 --snapshot=false

Backtrace

2022-09-05T11:06:15.337+09:00 | ERROR[09-05\|02:06:15.337] Error in block freeze operation err="block receipts missing, can't freeze block 15377501"
-- | --
  | 2022-09-05T11:06:32.260+09:00 | INFO [09-05\|02:06:32.260] Imported new chain segment blocks=1 txs=363 mgas=29.988 elapsed=1.759s mgasps=17.039 number=15,475,174 hash=6c49d8..a7d27a age=1m39s dirty=255.80MiB
  | 2022-09-05T11:06:32.267+09:00 | INFO [09-05\|02:06:32.267] Unindexed transactions blocks=1 txs=100 tail=13,125,175 elapsed=6.674ms
  | 2022-09-05T11:07:01.776+09:00 | INFO [09-05\|02:07:01.776] Imported new chain segment blocks=1 txs=326 mgas=29.968 elapsed=416.170ms mgasps=72.008 number=15,475,175 hash=9d7feb..32b5ad dirty=254.61MiB
  | 2022-09-05T11:07:01.784+09:00 | INFO [09-05\|02:07:01.784] Unindexed transactions blocks=1 txs=326 tail=13,125,176 elapsed=7.418ms
  | 2022-09-05T11:07:08.136+09:00 | INFO [09-05\|02:07:08.136] Imported new chain segment blocks=1 txs=329 mgas=29.980 elapsed=405.613ms mgasps=73.912 number=15,475,176 hash=d062ad..1f1bea dirty=254.11MiB
  | 2022-09-05T11:07:08.140+09:00 | INFO [09-05\|02:07:08.139] Unindexed transactions blocks=1 txs=157 tail=13,125,177 elapsed=3.309ms
  | 2022-09-05T11:07:15.340+09:00 | ERROR[09-05\|02:07:15.340] Error in block freeze operation err="block receipts missing, can't freeze block 15377501"
  | 2022-09-05T11:07:21.936+09:00 | INFO [09-05\|02:07:21.936] Imported new chain segment blocks=1 txs=416 mgas=29.980 elapsed=754.479ms mgasps=39.736 number=15,475,177 hash=7aa420..188263 dirty=254.53MiB
  | 2022-09-05T11:07:21.940+09:00 | INFO [09-05\|02:07:21.940] Unindexed transactions blocks=1 txs=115 tail=13,125,178 elapsed=1.800ms
  | 2022-09-05T11:07:33.367+09:00 | INFO [09-05\|02:07:33.366] Imported new chain segment blocks=1 txs=448 mgas=29.998 elapsed=1.049s mgasps=28.581 number=15,475,178 hash=63238e..e8564d dirty=254.92MiB
  | 2022-09-05T11:07:33.372+09:00 | INFO [09-05\|02:07:33.371] Unindexed transactions blocks=1 txs=227 tail=13,125,179 elapsed=4.633ms
  | 2022-09-05T11:07:35.624+09:00 | INFO [09-05\|02:07:35.624] Imported new chain segment blocks=1 txs=346 mgas=26.606 elapsed=565.968ms mgasps=47.010 number=15,475,179 hash=50bb6f..51e64f dirty=254.91MiB
  | 2022-09-05T11:07:35.627+09:00 | INFO [09-05\|02:07:35.627] Unindexed transactions blocks=1 txs=122 tail=13,125,180 elapsed=2.530ms
  | 2022-09-05T11:08:10.128+09:00 | INFO [09-05\|02:08:10.128] Imported new chain segment blocks=1 txs=281 mgas=30.044 elapsed=342.043ms mgasps=87.838 number=15,475,180 hash=60ab4c..65a6dd dirty=254.76MiB
  | 2022-09-05T11:08:10.131+09:00 | INFO [09-05\|02:08:10.131] Unindexed transactions blocks=1 txs=152 tail=13,125,181 elapsed=3.181ms
  | 2022-09-05T11:08:11.069+09:00 | INFO [09-05\|02:08:11.069] Imported new chain segment blocks=1 txs=185 mgas=19.539 elapsed=654.667ms mgasps=29.846 number=15,475,181 hash=64e629..14dcc9 dirty=255.60MiB
  | 2022-09-05T11:08:11.072+09:00 | INFO [09-05\|02:08:11.071] Unindexed transactions blocks=1 txs=58 tail=13,125,182 elapsed=1.484ms
  | 2022-09-05T11:08:15.343+09:00 | ERROR[09-05\|02:08:15.343] Error in block freeze operation err="block receipts missing, can't freeze block 15377501"

[rjl493456442]

The chain progressing is actually still going on. Another background thread for moving blocks from live database(leveldb) to ancient database(flat files) is blocked because of this missing receipts.

Also there is not enough information for us to know why it can happen. You can just ignore the error if you don't mind and the data freezing function will be disabled(so all the data will be put in the leveldb), but will recommend to resync if you can allow a few offline time.

[amplicity]

I'm having this same issue, but not sure of the best way to remove the geth db. I know there is geth removedb, but my data directory is in a custom location.

My geth startup command is:

./geth  --http --http.api eth,net,engine,admin --authrpc.jwtsecret ~/ethereum/my-jwt.hex --datadir='~/ethereum/execution/data';

If I want to get rid of this error, would I run ./geth --datadir='~/ethereum/execution/data' removedb ?

[MariusVanDerWijden]

Yes that should work to remove the db in a custom data directory

[suxnju]

Yes that should work to remove the db in a custom data directory

Hello, the latest block becomes 0 after removedb, is that normal?

My geth startup command is

geth --mainnet \
     --syncmode full \
     --gcmode archive \
     --http \
     --http.addr ${rpcaddr} \
     --http.vhosts "*" \
     --http.api "eth,net,web3,debug,admin" \
     --http.port "8545" \
     --http.corsdomain "*" \
     ...

If it is normal, I want to know when this block will start downloading?

[MariusVanDerWijden]

Yes its normal that the latest block becomes 0 after you do a removedb. The latest block is the newest block that you have the state for. Since you removed the state with removedb, the only state you have is for the genesis block. The node will start downloading once it has enough peers, but usually within a couple of minutes.
Closing this issue, feel free to open a new one if you encounter this error