Ethereum wallet will not sync past the last 65 blocks

[radu123ca]

My Ethereum wallet will not sync past the last 65 blocks. it always gets suck on the last 65 blocks.
i am running windows 10 64 bit and using Geth 1.8.2

• I've asked for help in the Mist Gitter before filing this issue.

Version: `0.0.0`
OS & Version: windows/linux/osx
Node version: `geth 0.0.0` 
Number of blocks synchronized: 0

[alexvandesande]

Is it always 65 blocks? It's common to geth to stall a bit in the last 1000 blocks for technical reasons. Are you using light client?

Sorry about the trouble syncing. This is typically an issue of geth taking a while to download many state entries before it can finalize the last blocks. If you haven’t tried syncing in Light mode, please do so by selecting the “Sync with Light client (beta)” option under the “Develop” menu tab. The feature is still in beta, but if you run into any trouble, you can usually clear it up by restarting.

If you have a time-sensitive need to move funds from your wallet, you can always access your wallet and send transactions within myetherwallet or mycrypto

We are merging all sync issues on this thread: #3097

[radu123ca]

Thank you for the reply and your time.

it always gets stuck 65 blocks away.
It does work with Light client (beta)
I managed to get it to work all by itself about a year ago on a slower machine (older mac book pro)
I'm using Version 0.9.3
I have not cleared the chain data and started over.

I think there is something wrong with the blockchain, does not share the last 65 blocks with new wallets (or maybe just my wallet)
I see a lot of people are having similar issues, and are told to try another non native wallet fix, coinbase, myetherwallet, etc.

[vampcheah]

i have the same trouble. i did removedb and resync, it's give me same result.

[radu123ca]

does yours vampcheah also stop exactly on the last 65 blocks?

[vampcheah]

yeah, always get 65 behind from the latest block.

[xen]

This is my context:

I found that I have the same problem. But also I noticed that number of downloading blocks decrease occasionally. After client restart it count until getting stuck on 65 missing blocks, but then can drop last few hundreds blocks and then again trying to catch chain structure.

But never getting into state when local chain structure in sync.

Upd. I tried on both macOS and Ubuntu Linux, and have the same problem.

[qwert76]

i have the same trouble. 65 behind from the latest block.

[CryptoBrainiac]

The same situation is constant, it does not synchronize the last 65 blocks from the entire blockchain. What is the reason for this?

Correction: now when there were 65 blocks left, I launched the application and switched to a light version, restarted the application and it synchronized all the blocks, now it works correctly on the light version.

https://i.gyazo.com/f1e551d86788208299a9d9b1a0fcb9ba.png

[jimihoffa]

Exactly same problem for me as well. 65 left always. occasionally it recognizes new blocks, then starts downloading blocks, only to stop when it is again 65 blocks left. Been at this for days. I've deleted chaindata (60gb) 3 times and completely redownloaded. I've tried removing node data as well. I've reinstalled wallet and geth. The same result everytime. I download 60gb of data, it hangs at 65 blocks left, and never ever completes.

[xen]

@CryptoBrainiac thank you, I switched to light with geth --light command. It works as a temporary workaround. Also, it saves a lot of space.

[radu123ca]

So as we stand now, there is no fix to this issue. Our only work around is to go for the light (beta) client?

[cryptocaterer]

Same issue stuck at last 65. Anyone found a fix yet?

[ptxd]

Problem affecting both Windows and Linux machines too. Ethereum sent to light client(main network) not displaying on main network. Temp work around is to switch between Test Net and Main Net repeatedly till synching is complete. No solution for ethereum sent to light client(main network) but not displaying on main network.

[Rocco768]

Ethereum-wallet, nanopool, claymore on Win10

Stuck at 65 blocks consistently, synchs new blocks and stops again at 65. Starting geth manually and and have deleted chaindata and resynch'd --fast.

Has there been any fix for this? or still working on it. My wallet still shows 0.00 but account info is correct

[cryptocaterer]

I uninstalled Geth and reinstalled it, it worked was able to create a few contracts tried two different methods. Both are still being created its been 48hrs for the first and 0 confirmations.

1st one - https://etherscan.io/tx/0x14bb22c7c4306743e7c0678fce2d1097ae0d183838979874b3b03c91f8f74166

2nd - https://etherscan.io/address/0x8355f4ef19eb8bad165b1386cffc8ed2ac8cfb37

If anyone has any advice please share.

[Vislor]

Same here, stuck at 65 blocks remaining.

[m0rani]

Hello. Same problem. Constantly stuck on 65 blocks.

[philsmd]

Well, first of all this is not really a mist (or ethereum wallet) problem, but an overall network protocol problem when using --fast sync. go-ethereum (geth) for which mist is basically a GUI for has exactly the same problems!
See: ethereum/go-ethereum#16251 (comment)
(within that post there is even a question for the 64/65 block misconception: "I'm stuck at 64 blocks behind mainnet?!")

So basically the problem is that the ethereum network protocol doesn't really have a clear overall picture of the maximum number of state tries.
Downloading the few blocks normally will be finished pretty quickly with the fast sync mode (and it doesn't really matter too much for the initial sync if it is 0 blocks behind or if there are "just" 64 blocks left, that's just a tiny number of bytes, that's not the problem at all in general), but this is just the very beginning of the syncing phase...
After having "almost all" the blocks, the node needs to basically (very oversimplified!) know the balances of the different addresses and therefore need to download the state tries to verify all the state tries/transactions/balances. This will in general take significantly much longer than just downloading few blocks :(

Again, you can read the whole details here: ethereum/go-ethereum#16251 (comment)

So yeah, if you want to have a "full node" (with fast sync instead of a light node) you basically need to wait very, very, very long for your node to download and verify all state tries. Your node won't know how many state tries exist (because the protocol doesn't broadcast this information) unless it is fully synced and doesn't receive any further state tries (which in theory will basically never happen because the network will always receive new transactions/contracts and therefore it will always try to catch up).
... but after your node synced all the state tries up to this 64/65 blocks left, it will basically change to a full node and will act like a full node without the --fast sync option (as far as I understood).
Therefore, you need to have a lot of patience (or use light node) and you shouldn't get distracted by these 64/65 blocks left (the blocks themself are normally not the actual problem!).

[m0rani]

Well, I understand, thanks for your answer.
I reserve my patience and I wait for loading.

[Xealott]

@philsmd
Thanks for some explanation. This has been driving me bonkers, nice to see it isn't just me.
I'm still learning the ins and outs of Ethereum... Are you saying that there is no way around this? All new adopters of a full node will have to contend with this "hung on last 65 blocks" issue? If so, it would be nice to include a note about this in the documentation...

What's the most appropriate forum for me to make this suggestion?

Thanks,
X

edit: have read @karalabe 's detailed explanation. The last question is the key... apparently it is unfeasible to sync a full node with an HDD, which is exactly what I was trying to do, calling my older machine into service. I think a lot of people will try to do this, so if this explanation is indeed accurate I still think it would be very helpful to the community to put this disclaimer up front ("only run light client on HDD!").
Again, any idea as to where I can most effectively post this suggestion welcome!

[Glorialiujie]

@alexvandesande After selecting the “Sync with Light client (beta)” , it successfully sync. However，it was stuck at “ethereum node connected” the next time I start mist and can not launch application.

[ChristBKK]

my problem is the same. Stuck for days on the last blocks.

Light Beta client is not really a solution as when I try to import the wallet contract it keeps me in an "importing..." loop. So still 0.10 wasn't the solution? Common Ethereum team it has been reported for months now that we have these problems.

[Glorialiujie]

I have solved it by running mist on ubuntu system.it can sync successfully in light beta client

[vampcheah]

so the final solution is light mode?

[philsmd]

it depends. If you really want to have a full node, you won't be satisfied with a light node... but you need to keep in mind that to operate a full node you need to have a fast SSD (solid state disk) and a good internet connection and also enough computing power and memory (RAM), otherwise the sync will take very long and be very painful (and with only a HDD it might never finish syncing)

[netbiz777]

Similar problem
The coins sent to the wallet are not visible
Windows 7, 64 bit

[valentinblehaut]

Same issue for me, stuck on 65 blocks remaining. Currently syncing on 'light' with 1 peer.

[onepremise]

I'm experiencing this problem with the latest geth client, 1.8.6. Using commandline options geth --sync-mode "fast" --cache 1024. It's been syncing on my mac high sierra laptop for the past several days, trailing the last 100 blocks.

[lorezyra]

Related: #2466

Please define OS and version.

[michaelgracie]

NOTE: posted here as well -> #3760

This issue has been replicated with both an external HDD and internal SSD storage. In either circumstance the block count sits 65 behind, with chain structure downloading in perpetuity. Watched for roughly an hour, and the blocks do update, simply reporting 65 behind no matter the storage device. ~20 peers either way.

Mist (Ethereum Wallet) version 0.10.0
macOS 10.13.4, MBP 2.5 GHz Intel Core i7, 16GB RAM
External storage: 1TB 7200rpm HDD, USB 3
Internal storage: 0.5TB stock SSD
Consistent 250 Mbps connection

Additionally tested simple wallet contract. Goes into creation mode, then after some time - presumably after another 65 blocks - comes up for confirmation. Cancelled thereafter.

[SpinnerRow]

I am having the same problem with 0.10.0 stuck at 65 blocks left. I have ETH in no mans land and need to get a resolution to this issue?

How can we escalate this?

[SpinnerRow]

After doing some more research and seeing this problem mentioned in several more GIT issues and in forums I decided to bail out of this wallet. It is a joke how long this has been going on with no resolution. Is there even a dev team for this project? Geeez!

[karalabe]

Syncing Ethereum is a pain point for many people, so I'll try to detail what's happening behind the scenes so there might be a bit less confusion.

The current default mode of sync for Geth is called fast sync. Instead of starting from the genesis block and reprocessing all the transactions that ever occurred (which could take weeks), fast sync downloads the blocks, and only verifies the associated proof-of-works. Downloading all the blocks is a straightforward and fast procedure and will relatively quickly reassemble the entire chain.

Many people falsely assume that because they have the blocks, they are in sync. Unfortunately this is not the case, since no transaction was executed, so we do not have any account state available (ie. balances, nonces, smart contract code and data). These need to be downloaded separately and cross checked with the latest blocks. This phase is called the state trie download and it actually runs concurrently with the block downloads; alas it take a lot longer nowadays than downloading the blocks.

So, what's the state trie? In the Ethereum mainnet, there are a ton of accounts already, which track the balance, nonce, etc of each user/contract. The accounts themselves are however insufficient to run a node, they need to be cryptographically linked to each block so that nodes can actually verify that the account's are not tampered with. This cryptographic linking is done by creating a tree data structure above the accounts, each level aggregating the layer below it into an ever smaller layer, until you reach the single root. This gigantic data structure containing all the accounts and the intermediate cryptographic proofs is called the state trie.

Ok, so why does this pose a problem? This trie data structure is an intricate interlink of hundreds of millions of tiny cryptographic proofs (trie nodes). To truly have a synchronized node, you need to download all the account data, as well as all the tiny cryptographic proofs to verify that noone in the network is trying to cheat you. This itself is already a crazy number of data items. The part where it gets even messier is that this data is constantly morphing: at every block (15s), about 1000 nodes are deleted from this trie and about 2000 new ones are added. This means your node needs to synchronize a dataset that is changing 200 times per second. The worst part is that while you are synchronizing, the network is moving forward, and state that you begun to download might disappear while you're downloading, so your node needs to constantly follow the network while trying to gather all the recent data. But until you actually do gather all the data, your local node is not usable since it cannot cryptographically prove anything about any accounts.

If you see that you are 64 blocks behind mainnet, you aren't yet synchronized, not even close. You are just done with the block download phase and still running the state downloads. You can see this yourself via the seemingly endless Imported state entries [...] stream of logs. You'll need to wait that out too before your node comes truly online.

---

Q: The node just hangs on importing state enties?!

A: The node doesn't hang, it just doesn't know how large the state trie is in advance so it keeps on going and going and going until it discovers and downloads the entire thing.

The reason is that a block in Ethereum only contains the state root, a single hash of the root node. When the node begins synchronizing, it knows about exactly 1 node and tries to download it. That node, can refer up to 16 new nodes, so in the next step, we'll know about 16 new nodes and try to download those. As we go along the download, most of the nodes will reference new ones that we didn't know about until then. This is why you might be tempted to think it's stuck on the same numbers. It is not, rather it's discovering and downloading the trie as it goes along.

Q: I'm stuck at 64 blocks behind mainnet?!

A: As explained above, you are not stuck, just finished with the block download phase, waiting for the state download phase to complete too. This latter phase nowadays take a lot longer than just getting the blocks.

Q: Why does downloading the state take so long, I have good bandwidth?

A: State sync is mostly limited by disk IO, not bandwidth.

The state trie in Ethereum contains hundreds of millions of nodes, most of which take the form of a single hash referencing up to 16 other hashes. This is a horrible way to store data on a disk, because there's almost no structure in it, just random numbers referencing even more random numbers. This makes any underlying database weep, as it cannot optimize storing and looking up the data in any meaningful way.

Not only is storing the data very suboptimal, but due to the 200 modification / second and pruning of past data, we cannot even download it is a properly pre-processed way to make it import faster without the underlying database shuffling it around too much. The end result is that even a fast sync nowadays incurs a huge disk IO cost, which is too much for a mechanical hard drive.

Q: Wait, so I can't run a full node on an HDD?

A: Unfortunately not. Doing a fast sync on an HDD will take more time than you're willing to wait with the current data schema. Even if you do wait it out, an HDD will not be able to keep up with the read/write requirements of transaction processing on mainnet.

You however should be able to run a light client on an HDD with minimal impact on system resources. If you wish to run a full node however, an SSD is your only option.

[jimihoffa]

This is excellent information, however some counter point.

I posted above about this issue way back on Mar 12. I had the same issues others were reporting. Stalling at 65 blocks. I have since abandoned attempting to sync due to this issue.

To be clear, it doesn't stall. It continues to download blocks. Constantly moving forward, while always maintaining 65 blocks of separation.

In response to your comments I allowed the sync to run close to 2 weeks at one time. Many other times I attempted shorter sessions, but tried a bunch of different sync methods, settings changes, etc that I found searching for the issue online. It was always the same. 65 blocks of separation.

A few points to note, disk IO. I would see spikes to IO, but never a constant stream once it halts at the separation of 65 blocks. The system I was attempting this on is a running 2 different drive setups. I tried on both. The first, which is my boot drive, is a Samsung 950 PRO nvme 512gb SSD. The second, which is the datastore for the system is a 4 drive raid 10 Samsung 850 PRO 512GB(x4). IO definitely isn't a concern. Connection is a 1gb down/300mb up cable connection, but that, as you mentioned, doesn't really matter.

I also had this working on a test system when I was doing my initial setup for my mining rig. It was fully synced NOT as a light client. My storage was similar, I was actually running a similar raid10 but with 4 different SSDs (lower performance at the time). There was a version change in-between rebuilding that system, and attempting to run the full wallet. Something changed. It may even be my fault, or our fault, but clearly multiple people are seeing this exact same 65 block separation. And as I mentioned, I can't speak for others, but I attempted sync on a system with high IO availability (and performing no other tasks), a fast connection, and almost a solid 2 weeks to complete sync (and multiple other attempts trying various methods after flushing and re-downloading 60gb+ multiple times).

Quick edit: Windows 10, apologies I don't know the geth version, but my initial posting was Mar 12. And I know there was an update prior to that at some point, so I would guess 1.8.1 and 1.8.2 with 1.8.2 being the version where I started having the issue, as I did have it working on the previous system on 1.8.1.

[lorezyra]

@karalabe,
I have this issue as well despite using a 2TB SSD external drive over USB-C. I doubt the issue is the throughput of the USB bus nor the bandwidth of the SSD processing speed. It's absolutely ridiculous to expect downloading the entire state tree before you can do anything with your Ethereum... Been waiting for almost a month now for my wallet to finish sync. I'm ready to delete the entire thing and move my tokens to a different crypto-coin...

Why is this not an issue with Bitcoin?? I can run the Bitcoin wallet software on my MBP2017 and it will finish the sync within 12 hours...

[michaelgracie]

Checked geth release dates, noting 1.8.2 was released Mar 5 and this particular issue opened three days hence. However, a similar issue has been opened again here -> ethereum/go-ethereum#16768 with 1.8.1-stable (released Feb 19).

Considering the numbers who have pointedly stated they are using SSDs, both external AND INTERNAL, it is probably safe to categorize the statement "you can't run a full node on an HDD" as a non sequitur.

UPDATE 16:41 MDT
Additional notes ...

• Ropsten (testnet) exhibits the same "65" behavior as Main
• Rinkeby syncs without issue

The former is roughly 13 GB while the latter is 8 GB, as of this update; I mention in case scale comes up. Both tests acquired a decent number of peers, quickly (much as happens with Main).

[evertonfraga]

I am closing this issue due to inactivity. Please feel free to reopen should you still have some considerations about it.

[hayorov]

I wrote a tiny python script to overview the process. It's here https://github.com/hayorov/ethereum-sync-mertics

My output:

2019-05-06 01:00:32 avg: 1827 max: 1938 min: 1378 states/s 	remain: 136604075 states	 4 peers 	eta@ 20:46:28.165828
2019-05-06 01:00:37 avg: 1864 max: 1938 min: 1378 states/s 	remain: 136595500 states	 3 peers 	eta@ 20:21:14.951050
2019-05-06 01:00:42 avg: 1791 max: 1938 min: 1378 states/s 	remain: 136583359 states	 3 peers 	eta@ 21:11:16.481006
2019-05-06 01:00:48 avg: 1742 max: 1938 min: 1378 states/s 	remain: 136580287 states	 3 peers 	eta@ 21:46:35.797305
2019-05-06 01:00:53 avg: 1721 max: 1938 min: 1378 states/s 	remain: 136575694 states	 3 peers 	eta@ 22:03:01.154434
2019-05-06 01:00:58 avg: 1682 max: 1938 min: 1378 states/s 	remain: 136569043 states	 4 peers 	eta@ 22:33:15.402442
2019-05-06 01:01:03 avg: 1698 max: 1938 min: 1378 states/s 	remain: 136564293 states	 3 peers 	eta@ 22:20:27.458747