Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Won't sync past block 516599 #268

Closed
ircrp opened this issue May 20, 2023 · 26 comments
Closed

Won't sync past block 516599 #268

ircrp opened this issue May 20, 2023 · 26 comments

Comments

@ircrp
Copy link

ircrp commented May 20, 2023

XDPosChain sync from scratch appears to be impossible at the moment due to masternode list mismatch at block 516600. This is the fourth time I am wiping & re-initializing the datadir from genesis and the error always arises on the same block. Note that up until block 516599 everything syncs fine and I have ensured that I am not syncing on a fork since the block hash is 0xa8731de5cf570a51305ab1df5dd84ad488b794179b7d2f80982d3c9d6227b91a.

Steps to reproduce:

git clone https://github.com/XinFinOrg/XDPoSChain.git
git checkout v1.4.6
wget https://raw.githubusercontent.com/XinFinOrg/XinFin-Node/master/mainnet/genesis.json
make XDC
./build/bin/XDC --datadir /home/me/.xinfin init genesis.json
./build/bin/XDC --datadir /home/me/.xinfin --networkid 50 --etherbase 0x000000000000000000000000000000000000dead

Error:

INFO [05-20|09:42:55] Imported new chain segment               blocks=449 txs=573 mgas=43.961 elapsed=259.648ms mgasps=169.311 number=516599 hash=a8731d…27b91a cache=1.40mB
ERROR[05-20|09:42:55] Masternodes lists are different in checkpoint header and snapshot number=516600 masternodes_from_checkpoint_header="...."
@gzliudan
Copy link
Collaborator

You can download snapshot from https://download.xinfin.network/xdcchain.tar, then synchronize chain from snapshot.

@ircrp
Copy link
Author

ircrp commented May 24, 2023 •
edited
Loading

You can download snapshot from https://download.xinfin.network/xdcchain.tar, then synchronize chain from snapshot.

Hi @gzliudan thanks for suggesting a workaround, however this yields a very similar error:

INFO [05-24|11:47:59] Updating new set of masternodes
INFO [05-24|11:47:59] New set of masternodes has been updated to snapshot number=59173650 hash=441e3f…8866cc new masternodes="[xdcB1693d224d4FD70aD6c9C9bDc44faa3ce21Fb40D xdc2a591F3d64F3cE6b1d2aFEEaD839ad76aAB9FEB2 xdc74D3aC0EFc4c22EA05150D7501c95C610b130c4B xdc047Ffe1fc7f6D0b7168C4CCc312221089629f470 xdc5651290BdD3a952357066B324F9334B544100140 xdc49D3FA92eEC838f644e8cACf2C93533d29b6C713 xdc90c87C9ff588E9268a1C7c79A5986886E98C2F04 xdc9043Fafbe8833ADafb449b559cEf300A9632763f xdcDb2E141595d8Edf6B1CE40b10d57Ca5b83855ec5 xdc609cc32E7513135349fbe9Ebd78EA8cdcFd9f819 xdc014428E60FaA874f8266D0648c52a83506DD9ec6 xdc48570E5DE69Db5203d259BBde6722BB1d3C3eE5a xdc70916660766043c23443479658008eEEdAD8Cb94 xdcEC4A5Fbd2e46E97bE98c212e668Ef1d08D695440 xdcbD705D4ea370ab75aF6622336aFb7B685c64A751 xdcDeF4bD67f9e2627Ca495b5f71794faB1bAb40619 xdc4e1b30F88C19786828f19b669881AE269DFEe57F xdc20d6826Ef6a151e327F5b5FC41E42fFF08C07311 xdcAf3dB8Ad8154820303ADc6be0D9B10ef8E33024F xdcccAb2F5267bc66b69f1688560C2E075216908A2C xdc1Aff171caa8C1ea93BAc4C27676f356D054412B4 xdc95907Ec85a9533a9622F079EF9E22f542Fae4c20 xdceD69a52eC69c872C54EC56828c5daF07174E74f3 xdc450d714e65F4dE007937C56Fb1c4686EfD4FB4EC xdc9a3787688fD210EC8F8D0224c6c50B8178d75Bc0 xdc39eE018a85e4469a1b335EFe014067E315bd6594 xdc241ff0d3096E2e0B477780f9f551918a06827c05 xdcD22fDAc1459760f698618D927BBe22249e2B29B9 xdcE3bC38418f89C386d1093aee89d606564012e329 xdc2f86476fAA31C3F5A3d5B9376282e1B02b4Dfa00 xdc35898873C021f9f4Fa01a76C7FA595159f5523E7 xdc3056a8bFf9A17b12D08F1837D0b44Cf1E2018fBe xdcB38ABA47A5563fa4AeDC2a649EF819b9FB160b6a xdc72fb467Ef6dA285b6B0D8f7A25AbD6049385d5fD xdce2BC9A03D5AE35E130C8bF99eA50c022de375DB7 xdc81b2EF52Cc6ba6c046Af701d94217cdbbD2746D5 xdc1E0103Ba7665D15328b3886927D4F0a85F8b2299 xdce865a5b2bf699A4b498De8a8c55Da14bB0d94b21 xdcE8E8194c0a8E5A32306C7A86052A4E8cA4B8729d xdc1088C53E04D139A20A2f8998C586d27139a89A9B xdccaEB8bEf9689E70135023d4dFb85864832f968BF xdc065551F0dcAC6f00CAe11192D462db709bE3758c xdc9FCe52C5C451599235c17bC37E99f846d25Ee6eC xdc45745Cd2376c452f9f44f64BcD82acE0cdA22546 xdc0B0aFb41c14dD921a9a4511c226dC29254Ba0d8F xdca70b87C39A1237Ba53953a1da04b23D4db28eACe xdc8f2Fb5Da850042B7DA5097061F098493F8Ec6dD8 xdcA72Ce94A09db26Dce57A4852409ABC2Fff07A962 xdc72a0224245992792C75565bD3ab6cAA6C8B95218 xdcc67c2DEc79da735d6587d8DB3c23271d557196ab xdcBbD2D417a8B6f1b1d7a267Cd1D7402b443F35CFE xdcb29B17b0a881d9da1F6976E4f22f8277d41Bc065 xdcdccd99DA1c942A36c5dcaf26D19B98C815DFcb73 xdcC3e2A9A73881FCCe6fc9367a0A3DdBD658d20E37 xdc25c65b4b379ac37CF78357c4915f73677022eaFF xdc22B2f4C9D6a1b7958b7315a25328984ba9A9D050 xdcA4657c02208797985AdEDCbd048efC82291Dbdb2 xdc497c44F4A22099109Ad7d194AaD4fBe78dCeB788 xdc03d9e17Ae3fF2c6712E44e25B09Ac5ee91f6c9ff xdc35943387085224cd07Efab4264E850cE6cdC99ac xdc8E455Bb521212bdcf8Cb326e32Dc1183cb3Fd887 xdc1A7c3D7481aDC7E067F2f73F35f44143d7bf02d4 xdcEf3C018667Aa871C02d3d6bE10a9a8cCdd351294 xdcE230905C99Aaa7b68402Af8611B89cedA743191E xdc1D25a7076993cfCd676237daea90D8EdC99C7F46 xdc24C0832d9df8189166d8e4e2ceDC40ad69077b17 xdcb0CC3fD7fE1df0f3D2d962746725f01693c49B31 xdc4B7076c988dA8a0eF87f1af137f7ABC39557b746 xdc1D393817AB218EBadE3e8ad591593EC3b132b1F6 xdcf04f32c46F1A16663bc7f6409B35b33F3DAa9a03 xdc65c90c2d3e99D8366f90db7F4D25f23A0a569d49 xdc52825E085aE7785AD9B42AAa98a630560593b7E2 xdcc98159feD7eE13D769f36D894E56db4061900485 xdcE494Fe5a38B7212AB999152D1d45ADAb8d84a8A7 xdc18d31511b8efbF4215c48C026c56aD469cE90374 xdcC02aed857b01b4d60b378096221dB3b60AfaDc37 xdc360B9E5870Fc91e375f902Fc134f91739f4A99c7 xdc7525f63E5F739EF952fbEA50C1dC97ef5d5BD984 xdcd70ef43752571BA7dFf8516863A12b7A26b3b45e xdc8B73C11E7b8e0Cae2B00E425d406D5c6aB62ee0e xdcD5BB37763625Bc16f2aD0791d87E2726004241e4 xdc153228715189a6317F7adBEd4cc6cb5Fa7D8b6Df xdcb1EFD51De35f2966c770Cd35B953f4FDADE3c3E8 xdcb9a3a97F6A02a86483bf02cB33F8B33d2D117708 xdc2d74d0125982bdC3A9F54A10216D82509379E821 xdcCBA3656A88647B475650CC000672adddAfC21b5D xdc3f0A77314CF0d9609fBf06B31C9Dc50959d8AC7c xdccABce70042847e347c22a1e32eEc85A56d78fceD xdc296F771674987B4d465E3563dAf2A75bfce811b6 xdcC7d49d0a2CF198DeEBd6ce581aF465944ec8b2bB xdc664c4a7B15d91B07c468162f535909114C038b91 xdccfCCDeA1006a5CfA7d9484B5b293B46964c265c0 xdc9C54bc764c2E00717fbBF31B4dc962a72c47bdf6 xdc63443fFdB5c139C3bBAb97D3b06ee0674d75AB7f xdc4E111142fbf2eDAcc4ab122fEB54b031DC3D95d9 xdc4398241671b3Dd484FE3213A4fB7511f30e7D7C0 xdc92D32364313C376Dcae272e113Fd04CEa194D2c8 xdc595b8170eAF2e53e47cc20db47aD063a1e6e2C0f xdc619f838eA2A12Cdb508E759c3E0697E021D52cEB xdc7aA125338BE075260e77c6a66A56c90A5DeC4C58 xdcE4710a854b24062fF37Ab6636bd9A456e24c1635 xdca65010026b83368Ca05DF6E8B467985D6de3eAC5 xdc08a8Ae21F20799944871039743316fb7B4aC72eF xdc90C60BD5A20bd2A3142e2c491B9a59C2C1E83E4D xdcD2FEfdD118aA2bEA5991dC079fd341D3e7a92cAF xdc1a1A8229e4a2Fa06f73A564092976ba2DFE6AAb8 xdcFC4a4D7E855B43A2bAD943559F164228a053739b xdcD6E9161fAd50d2E697CdA2b960fD26Bf52a5D169]"
INFO [05-24|11:47:59] Masternodes are ready for the next epoch
INFO [05-24|11:47:59] Imported new chain segment               blocks=112 txs=868 mgas=0.070  elapsed=1.131s    mgasps=0.062  number=59173650 hash=441e3f…8866cc cache=2.25mB
INFO [05-24|11:47:59] Imported new chain segment               blocks=16  txs=131 mgas=0.042  elapsed=37.315ms  mgasps=1.126  number=59173666 hash=95e162…bbb700 cache=2.27mB
INFO [05-24|11:47:59] Persisted trie from memory database      nodes=766 size=272.24kB time=1.893299ms gcnodes=4553 gcsize=1.70mB gctime=3.492138ms livenodes=4131 livesize=1.55mB
INFO [05-24|11:47:59] Persisted trie from memory database      nodes=0   size=0.00B    time=1.213µs    gcnodes=0    gcsize=0.00B  gctime=9.921µs    livenodes=1    livesize=0.00B
INFO [05-24|11:47:59] Persisted trie from memory database      nodes=0   size=0.00B    time=350ns      gcnodes=0    gcsize=0.00B  gctime=8.749µs    livenodes=1    livesize=0.00B
INFO [05-24|11:48:00] Persisted trie from memory database      nodes=547 size=202.44kB time=1.473932ms gcnodes=3583 gcsize=1.34mB gctime=2.554492ms livenodes=5677 livesize=2.07mB
INFO [05-24|11:48:00] Persisted trie from memory database      nodes=0   size=0.00B    time=1.188µs    gcnodes=0    gcsize=0.00B  gctime=9.524µs    livenodes=1    livesize=0.00B
INFO [05-24|11:48:00] Persisted trie from memory database      nodes=0   size=0.00B    time=222ns      gcnodes=0    gcsize=0.00B  gctime=8.624µs    livenodes=1    livesize=0.00B
INFO [05-24|11:48:00] Persisted trie from memory database      nodes=915 size=313.25kB time=2.282607ms gcnodes=4764 gcsize=1.76mB gctime=3.893314ms livenodes=7986 livesize=2.74mB
INFO [05-24|11:48:00] Persisted trie from memory database      nodes=0   size=0.00B    time=1.215µs    gcnodes=0    gcsize=0.00B  gctime=10.15µs    livenodes=1    livesize=0.00B
INFO [05-24|11:48:00] Persisted trie from memory database      nodes=0   size=0.00B    time=412ns      gcnodes=0    gcsize=0.00B  gctime=9.431µs    livenodes=1    livesize=0.00B
INFO [05-24|11:48:00] Imported new chain segment               blocks=384 txs=3594 mgas=28.348 elapsed=1.306s    mgasps=21.692 number=59174050 hash=6e217b…31eb0f cache=3.66mB
INFO [05-24|11:48:00] Imported new chain segment               blocks=49  txs=487  mgas=5.037  elapsed=139.824ms mgasps=36.021 number=59174099 hash=0945c1…7ea1b2 cache=3.44mB
ERROR[05-24|11:48:00] Masternodes lists are different in checkpoint header and snapshot number=59174100 masternodes_from_checkpoint_header="[[1 68 40 230 15 170 135 79 130 102 208 100 140 82 168 53 6 221 158 198] [3 217 225 122 227 255 44 103 18 228 78 37 176 154 197 238 145 246 201 255] [4 127 254 31 199 246 208 183 22 140 76 204 49 34 33 8 150 41 244 112] [8 168 174 33 242 7 153 148 72 113 3 151 67 49 111 183 180 172 114 239] [11 10 251 65 193 77 217 33 169 164 81 28 34 109 194 146 84 186 13 143] [14 38 165 75 247 12 81 70 12 177 63 72 35 2 118 120 103 46 74 10] [16 136 197 62 4 209 57 162 10 47 137 152 197 134 210 113 57 168 154 155] [21 50 40 113 81 137 166 49 127 122 219 237 76 198 203 95 167 216 182 223] [24 211 21 17 184 239 191 66 21 196 140 2 108 86 173 70 156 233 3 116] [26 124 61 116 129 173 199 224 103 242 247 63 53 244 65 67 215 191 2 212] [26 255 23 28 170 140 30 169 59 172 76 39 103 111 53 109 5 68 18 180] [29 37 167 7 105 147 207 205 103 98 55 218 234 144 216 237 201 156 127 70] [29 57 56 23 171 33 142 186 222 62 138 213 145 89 62 195 177 50 177 246] [30 1 3 186 118 101 209 83 40 179 136 105 39 212 240 168 95 139 34 153] [26 26 130 41 228 162 250 6 247 58 86 64 146 151 107 162 223 230 170 184] [32 214 130 110 246 161 81 227 39 245 181 252 65 228 47 255 8 192 115 17] [34 178 244 201 214 161 183 149 139 115 21 162 83 40 152 75 169 169 208 80] [36 31 240 211 9 110 46 11 71 119 128 249 245 81 145 138 6 130 124 5] [36 192 131 45 157 248 24 145 102 216 228 226 206 220 64 173 105 7 123 23] [37 198 91 75 55 154 195 124 247 131 87 196 145 95 115 103 112 34 234 255] [41 111 119 22 116 152 123 77 70 94 53 99 218 242 167 91 252 232 17 182] [42 89 31 61 100 243 206 107 29 42 254 234 216 57 173 118 170 185 254 178] [45 116 208 18 89 130 189 195 169 245 74 16 33 109 130 80 147 121 232 33] [47 134 71 111 170 49 195 245 163 213 185 55 98 130 225 176 43 77 250 0] [48 86 168 191 249 161 123 18 208 143 24 55 208 180 76 241 226 1 143 190] [53 137 136 115 192 33 249 244 250 1 167 108 127 165 149 21 159 85 35 231] [53 148 51 135 8 82 36 205 7 239 171 66 100 232 80 206 108 220 153 172] [54 11 158 88 112 252 145 227 117 249 2 252 19 79 145 115 159 74 153 199] [57 238 1 138 133 228 70 154 27 51 94 254 1 64 103 227 21 189 101 148] [63 10 119 49 76 240 217 96 159 191 6 179 28 157 197 9 89 216 172 124] [67 152 36 22 113 179 221 72 79 227 33 58 79 183 81 31 48 231 215 192] [69 13 113 78 101 244 222 0 121 55 197 111 177 196 104 110 253 79 180 236] [69 116 92 210 55 108 69 47 159 68 246 75 205 130 172 224 205 162 37 70] [72 87 14 93 230 157 181 32 61 37 155 189 230 114 43 177 211 195 238 90] [73 124 68 244 162 32 153 16 154 215 209 148 170 212 251 231 141 206 183 136] [73 211 250 146 238 200 56 246 68 232 202 207 44 147 83 61 41 182 199 19] [75 112 118 201 136 218 138 14 248 127 26 241 55 247 171 195 149 87 183 70] [78 17 17 66 251 242 237 172 196 171 18 47 235 84 176 49 220 61 149 217] [78 27 48 248 140 25 120 104 40 241 155 102 152 129 174 38 157 254 229 127] [82 130 94 8 90 231 120 90 217 180 42 170 152 166 48 86 5 147 183 226] [86 81 41 11 221 58 149 35 87 6 107 50 79 147 52 181 68 16 1 64] [89 91 129 112 234 242 229 62 71 204 32 219 71 173 6 58 30 110 44 15] [96 156 195 46 117 19 19 83 73 251 233 235 215 142 168 205 207 217 248 25] [97 159 131 142 162 161 44 219 80 142 117 156 62 6 151 224 33 213 44 235] [99 68 63 253 181 193 57 195 187 171 151 211 176 110 224 103 77 117 171 127] [101 201 12 45 62 153 216 54 111 144 219 127 77 37 242 58 10 86 157 73] [102 76 74 123 21 217 27 7 196 104 22 47 83 89 9 17 76 3 139 145] [112 145 102 96 118 96 67 194 52 67 71 150 88 0 142 238 218 216 203 148] [114 160 34 66 69 153 39 146 199 85 101 189 58 182 202 166 200 185 82 24] [114 251 70 126 246 218 40 91 107 13 143 122 37 171 214 4 147 133 213 253] [116 211 172 14 252 76 34 234 5 21 13 117 1 201 92 97 11 19 12 75] [117 37 246 62 95 115 158 249 82 251 234 80 193 220 151 239 93 91 217 132] [122 161 37 51 139 224 117 38 14 119 198 166 106 86 201 10 93 236 76 88] [129 178 239 82 204 107 166 192 70 175 112 29 148 33 124 219 189 39 70 213] [139 115 193 30 123 142 12 174 43 0 228 37 212 6 213 198 171 98 238 14] [142 69 91 181 33 33 43 220 248 203 50 110 50 220 17 131 203 63 216 135] [143 47 181 218 133 0 66 183 218 80 151 6 31 9 132 147 248 236 109 216] [144 67 250 251 232 131 58 218 251 68 155 85 156 239 48 10 150 50 118 63] [144 198 11 213 162 11 210 163 20 46 44 73 27 154 89 194 193 232 62 77] [144 200 124 159 245 136 233 38 138 28 124 121 165 152 104 134 233 140 47 4] [146 211 35 100 49 60 55 109 202 226 114 225 19 253 4 206 161 148 210 200] [149 144 126 200 90 149 51 169 98 47 7 158 249 226 47 84 47 174 76 32] [156 84 188 118 76 46 0 113 127 187 243 27 77 201 98 167 44 71 189 246] [159 206 82 197 196 81 89 146 53 193 123 195 126 153 248 70 210 94 230 236] [154 55 135 104 143 210 16 236 143 141 2 36 198 197 11 129 120 215 91 192] [164 101 124 2 32 135 151 152 90 222 220 189 4 142 252 130 41 29 189 178] [167 44 233 74 9 219 38 220 229 122 72 82 64 154 188 47 255 7 169 98] [175 61 184 173 129 84 130 3 3 173 198 190 13 155 16 239 142 51 2 79] [177 105 61 34 77 79 215 10 214 201 201 189 196 79 170 60 226 31 180 13] [179 138 186 71 165 86 63 164 174 220 42 100 158 248 25 185 251 22 11 106] [187 210 212 23 168 182 241 177 215 162 103 205 29 116 2 180 67 243 92 254] [192 42 237 133 123 1 180 214 11 55 128 150 34 29 179 182 10 250 220 55] [195 226 169 167 56 129 252 206 111 201 54 122 10 61 219 214 88 210 14 55] [199 212 157 10 44 241 152 222 235 214 206 88 26 244 101 148 78 200 178 187] [203 163 101 106 136 100 123 71 86 80 204 0 6 114 173 221 175 194 27 93] [210 47 218 193 69 151 96 246 152 97 141 146 123 190 34 36 158 43 41 185] [210 254 253 209 24 170 43 234 89 145 220 7 159 211 65 211 231 169 44 175] [213 187 55 118 54 37 188 22 242 173 7 145 216 126 39 38 0 66 65 228] [219 46 20 21 149 216 237 246 177 206 64 177 13 87 202 91 131 133 94 197] [222 244 189 103 249 226 98 124 164 149 181 247 23 148 250 177 186 180 6 25] [226 48 144 92 153 170 167 182 132 2 175 134 17 184 156 237 167 67 25 30] [227 188 56 65 143 137 195 134 209 9 58 238 137 214 6 86 64 18 227 41] [228 113 10 133 75 36 6 47 243 122 182 99 107 217 164 86 226 76 22 53] [228 148 254 90 56 183 33 42 185 153 21 45 29 69 173 171 141 132 168 167] [232 232 25 76 10 142 90 50 48 108 122 134 5 42 78 140 164 184 114 157] [236 74 95 189 46 70 233 123 233 140 33 46 102 142 241 208 141 105 84 64] [239 60 1 134 103 170 135 28 2 211 214 190 16 169 168 204 221 53 18 148] [166 80 16 2 107 131 54 140 160 93 246 232 180 103 152 93 109 227 234 197] [167 11 135 195 154 18 55 186 83 149 58 29 160 75 35 212 219 40 234 206] [176 204 63 215 254 29 240 243 210 217 98 116 103 37 240 22 147 196 155 49] [177 239 213 29 227 95 41 102 199 112 205 53 185 83 244 253 173 227 195 232] [178 155 23 176 168 129 217 218 31 105 118 228 242 47 130 119 212 27 192 101] [185 163 169 127 106 2 168 100 131 191 2 203 51 248 179 61 45 17 119 8] [189 112 93 78 163 112 171 117 175 102 34 51 106 251 123 104 92 100 167 81] [198 124 45 236 121 218 115 93 101 135 216 219 60 35 39 29 85 113 150 171] [201 129 89 254 215 238 19 215 105 243 109 137 78 86 219 64 97 144 4 133] [202 188 231 0 66 132 126 52 124 34 161 227 46 236 133 165 109 120 252 237] [202 235 139 239 150 137 231 1 53 2 61 77 251 133 134 72 50 249 104 191] [204 171 47 82 103 188 102 182 159 22 136 86 12 46 7 82 22 144 138 44] [207 204 222 161 0 106 92 250 125 148 132 181 178 147 180 105 100 194 101 192] [215 14 244 55 82 87 27 167 223 248 81 104 99 161 43 122 38 179 180 94] [220 205 153 218 28 148 42 54 197 220 175 38 209 155 152 200 21 223 203 115] [226 188 154 3 213 174 53 225 48 200 191 153 234 80 192 34 222 55 93 183] [232 101 165 178 191 105 154 75 73 141 232 168 197 93 161 75 176 217 75 33] [237 105 165 46 198 156 135 44 84 236 86 130 140 93 175 7 23 78 116 243] [240 79 50 196 111 26 22 102 59 199 246 64 155 53 179 63 61 170 154 3] [250 154 31 46 195 217 36 40 252 97 5 164 224 88 50 174 116 23 10 196]]" masternodes_in_snapshot="[[1 68 40 230 15 170 135 79 130 102 208 100 140 82 168 53 6 221 158 198] [3 217 225 122 227 255 44 103 18 228 78 37 176 154 197 238 145 246 201 255] [4 127 254 31 199 246 208 183 22 140 76 204 49 34 33 8 150 41 244 112] [8 168 174 33 242 7 153 148 72 113 3 151 67 49 111 183 180 172 114 239] [11 10 251 65 193 77 217 33 169 164 81 28 34 109 194 146 84 186 13 143] [16 136 197 62 4 209 57 162 10 47 137 152 197 134 210 113 57 168 154 155] [21 50 40 113 81 137 166 49 127 122 219 237 76 198 203 95 167 216 182 223] [24 211 21 17 184 239 191 66 21 196 140 2 108 86 173 70 156 233 3 116] [26 124 61 116 129 173 199 224 103 242 247 63 53 244 65 67 215 191 2 212] [26 255 23 28 170 140 30 169 59 172 76 39 103 111 53 109 5 68 18 180] [29 37 167 7 105 147 207 205 103 98 55 218 234 144 216 237 201 156 127 70] [29 57 56 23 171 33 142 186 222 62 138 213 145 89 62 195 177 50 177 246] [30 1 3 186 118 101 209 83 40 179 136 105 39 212 240 168 95 139 34 153] [26 26 130 41 228 162 250 6 247 58 86 64 146 151 107 162 223 230 170 184] [32 214 130 110 246 161 81 227 39 245 181 252 65 228 47 255 8 192 115 17] [34 178 244 201 214 161 183 149 139 115 21 162 83 40 152 75 169 169 208 80] [36 31 240 211 9 110 46 11 71 119 128 249 245 81 145 138 6 130 124 5] [36 192 131 45 157 248 24 145 102 216 228 226 206 220 64 173 105 7 123 23] [37 198 91 75 55 154 195 124 247 131 87 196 145 95 115 103 112 34 234 255] [41 111 119 22 116 152 123 77 70 94 53 99 218 242 167 91 252 232 17 182] [42 89 31 61 100 243 206 107 29 42 254 234 216 57 173 118 170 185 254 178] [45 116 208 18 89 130 189 195 169 245 74 16 33 109 130 80 147 121 232 33] [47 134 71 111 170 49 195 245 163 213 185 55 98 130 225 176 43 77 250 0] [48 86 168 191 249 161 123 18 208 143 24 55 208 180 76 241 226 1 143 190] [53 137 136 115 192 33 249 244 250 1 167 108 127 165 149 21 159 85 35 231] [53 148 51 135 8 82 36 205 7 239 171 66 100 232 80 206 108 220 153 172] [54 11 158 88 112 252 145 227 117 249 2 252 19 79 145 115 159 74 153 199] [57 238 1 138 133 228 70 154 27 51 94 254 1 64 103 227 21 189 101 148] [63 10 119 49 76 240 217 96 159 191 6 179 28 157 197 9 89 216 172 124] [67 152 36 22 113 179 221 72 79 227 33 58 79 183 81 31 48 231 215 192] [69 13 113 78 101 244 222 0 121 55 197 111 177 196 104 110 253 79 180 236] [69 116 92 210 55 108 69 47 159 68 246 75 205 130 172 224 205 162 37 70] [72 87 14 93 230 157 181 32 61 37 155 189 230 114 43 177 211 195 238 90] [73 124 68 244 162 32 153 16 154 215 209 148 170 212 251 231 141 206 183 136] [73 211 250 146 238 200 56 246 68 232 202 207 44 147 83 61 41 182 199 19] [75 112 118 201 136 218 138 14 248 127 26 241 55 247 171 195 149 87 183 70] [78 17 17 66 251 242 237 172 196 171 18 47 235 84 176 49 220 61 149 217] [78 27 48 248 140 25 120 104 40 241 155 102 152 129 174 38 157 254 229 127] [82 130 94 8 90 231 120 90 217 180 42 170 152 166 48 86 5 147 183 226] [86 81 41 11 221 58 149 35 87 6 107 50 79 147 52 181 68 16 1 64] [89 91 129 112 234 242 229 62 71 204 32 219 71 173 6 58 30 110 44 15] [96 156 195 46 117 19 19 83 73 251 233 235 215 142 168 205 207 217 248 25] [97 159 131 142 162 161 44 219 80 142 117 156 62 6 151 224 33 213 44 235] [99 68 63 253 181 193 57 195 187 171 151 211 176 110 224 103 77 117 171 127] [101 201 12 45 62 153 216 54 111 144 219 127 77 37 242 58 10 86 157 73] [102 76 74 123 21 217 27 7 196 104 22 47 83 89 9 17 76 3 139 145] [112 145 102 96 118 96 67 194 52 67 71 150 88 0 142 238 218 216 203 148] [114 160 34 66 69 153 39 146 199 85 101 189 58 182 202 166 200 185 82 24] [114 251 70 126 246 218 40 91 107 13 143 122 37 171 214 4 147 133 213 253] [116 211 172 14 252 76 34 234 5 21 13 117 1 201 92 97 11 19 12 75] [117 37 246 62 95 115 158 249 82 251 234 80 193 220 151 239 93 91 217 132] [122 161 37 51 139 224 117 38 14 119 198 166 106 86 201 10 93 236 76 88] [129 178 239 82 204 107 166 192 70 175 112 29 148 33 124 219 189 39 70 213] [139 115 193 30 123 142 12 174 43 0 228 37 212 6 213 198 171 98 238 14] [142 69 91 181 33 33 43 220 248 203 50 110 50 220 17 131 203 63 216 135] [143 47 181 218 133 0 66 183 218 80 151 6 31 9 132 147 248 236 109 216] [144 67 250 251 232 131 58 218 251 68 155 85 156 239 48 10 150 50 118 63] [144 198 11 213 162 11 210 163 20 46 44 73 27 154 89 194 193 232 62 77] [144 200 124 159 245 136 233 38 138 28 124 121 165 152 104 134 233 140 47 4] [146 211 35 100 49 60 55 109 202 226 114 225 19 253 4 206 161 148 210 200] [149 144 126 200 90 149 51 169 98 47 7 158 249 226 47 84 47 174 76 32] [156 84 188 118 76 46 0 113 127 187 243 27 77 201 98 167 44 71 189 246] [159 206 82 197 196 81 89 146 53 193 123 195 126 153 248 70 210 94 230 236] [154 55 135 104 143 210 16 236 143 141 2 36 198 197 11 129 120 215 91 192] [164 101 124 2 32 135 151 152 90 222 220 189 4 142 252 130 41 29 189 178] [167 44 233 74 9 219 38 220 229 122 72 82 64 154 188 47 255 7 169 98] [175 61 184 173 129 84 130 3 3 173 198 190 13 155 16 239 142 51 2 79] [177 105 61 34 77 79 215 10 214 201 201 189 196 79 170 60 226 31 180 13] [179 138 186 71 165 86 63 164 174 220 42 100 158 248 25 185 251 22 11 106] [187 210 212 23 168 182 241 177 215 162 103 205 29 116 2 180 67 243 92 254] [192 42 237 133 123 1 180 214 11 55 128 150 34 29 179 182 10 250 220 55] [195 226 169 167 56 129 252 206 111 201 54 122 10 61 219 214 88 210 14 55] [199 212 157 10 44 241 152 222 235 214 206 88 26 244 101 148 78 200 178 187] [203 163 101 106 136 100 123 71 86 80 204 0 6 114 173 221 175 194 27 93] [210 47 218 193 69 151 96 246 152 97 141 146 123 190 34 36 158 43 41 185] [210 254 253 209 24 170 43 234 89 145 220 7 159 211 65 211 231 169 44 175] [213 187 55 118 54 37 188 22 242 173 7 145 216 126 39 38 0 66 65 228] [214 233 22 31 173 80 210 230 151 205 162 185 96 253 38 191 82 165 209 105] [219 46 20 21 149 216 237 246 177 206 64 177 13 87 202 91 131 133 94 197] [222 244 189 103 249 226 98 124 164 149 181 247 23 148 250 177 186 180 6 25] [226 48 144 92 153 170 167 182 132 2 175 134 17 184 156 237 167 67 25 30] [227 188 56 65 143 137 195 134 209 9 58 238 137 214 6 86 64 18 227 41] [228 113 10 133 75 36 6 47 243 122 182 99 107 217 164 86 226 76 22 53] [228 148 254 90 56 183 33 42 185 153 21 45 29 69 173 171 141 132 168 167] [232 232 25 76 10 142 90 50 48 108 122 134 5 42 78 140 164 184 114 157] [236 74 95 189 46 70 233 123 233 140 33 46 102 142 241 208 141 105 84 64] [239 60 1 134 103 170 135 28 2 211 214 190 16 169 168 204 221 53 18 148] [252 74 77 126 133 91 67 162 186 217 67 85 159 22 66 40 160 83 115 155] [166 80 16 2 107 131 54 140 160 93 246 232 180 103 152 93 109 227 234 197] [167 11 135 195 154 18 55 186 83 149 58 29 160 75 35 212 219 40 234 206] [176 204 63 215 254 29 240 243 210 217 98 116 103 37 240 22 147 196 155 49] [177 239 213 29 227 95 41 102 199 112 205 53 185 83 244 253 173 227 195 232] [178 155 23 176 168 129 217 218 31 105 118 228 242 47 130 119 212 27 192 101] [185 163 169 127 106 2 168 100 131 191 2 203 51 248 179 61 45 17 119 8] [189 112 93 78 163 112 171 117 175 102 34 51 106 251 123 104 92 100 167 81] [198 124 45 236 121 218 115 93 101 135 216 219 60 35 39 29 85 113 150 171] [201 129 89 254 215 238 19 215 105 243 109 137 78 86 219 64 97 144 4 133] [202 188 231 0 66 132 126 52 124 34 161 227 46 236 133 165 109 120 252 237] [202 235 139 239 150 137 231 1 53 2 61 77 251 133 134 72 50 249 104 191] [204 171 47 82 103 188 102 182 159 22 136 86 12 46 7 82 22 144 138 44] [207 204 222 161 0 106 92 250 125 148 132 181 178 147 180 105 100 194 101 192] [215 14 244 55 82 87 27 167 223 248 81 104 99 161 43 122 38 179 180 94] [220 205 153 218 28 148 42 54 197 220 175 38 209 155 152 200 21 223 203 115] [226 188 154 3 213 174 53 225 48 200 191 153 234 80 192 34 222 55 93 183] [232 101 165 178 191 105 154 75 73 141 232 168 197 93 161 75 176 217 75 33] [237 105 165 46 198 156 135 44 84 236 86 130 140 93 175 7 23 78 116 243] [240 79 50 196 111 26 22 102 59 199 246 64 155 53 179 63 61 170 154 3]]" penList=[]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x75ad1b]

goroutine 36162 [running]:
github.com/XinFinOrg/XDPoSChain/core/state.(*StateDB).getStateObject(0x0, {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...})
        /home/me/dev/nodes/XDPoSChain/core/state/statedb.go:388 +0x7b
github.com/XinFinOrg/XDPoSChain/core/state.(*StateDB).GetState(0x0, {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...)
        /home/me/dev/nodes/XDPoSChain/core/state/statedb.go:256 +0x50
github.com/XinFinOrg/XDPoSChain/core/state.GetCandidates(0x60e81d4294136a52?)
        /home/me/dev/nodes/XDPoSChain/core/state/statedb_utils.go:95 +0x12c
github.com/XinFinOrg/XDPoSChain/eth/hooks.AttachConsensusV1Hooks.func5({0x44, 0x1e, 0x3f, 0x68, 0xb1, 0xeb, 0x4c, 0x7b, 0xd6, 0xde, ...})
        /home/me/dev/nodes/XDPoSChain/eth/hooks/engine_v1_hooks.go:231 +0x179
github.com/XinFinOrg/XDPoSChain/consensus/XDPoS/engines/engine_v1.(*XDPoS_v1).getSignersFromContract(0xc00019d6b0, {0x14b6f38, 0xc00afc2000}, 0xc00f6f6a00?)
        /home/me/dev/nodes/XDPoSChain/consensus/XDPoS/engines/engine_v1/engine.go:1003 +0x178
github.com/XinFinOrg/XDPoSChain/consensus/XDPoS/engines/engine_v1.(*XDPoS_v1).verifyCascadingFields(0xc00019d6b0, {0x14b6f38, 0xc00afc2000}, 0xc00f6f6a00, {0xc03992c208?, 0x0, 0x1}, 0xa0?)
        /home/me/dev/nodes/XDPoSChain/consensus/XDPoS/engines/engine_v1/engine.go:263 +0x365
github.com/XinFinOrg/XDPoSChain/consensus/XDPoS/engines/engine_v1.(*XDPoS_v1).verifyHeader(0xc00019d6b0, {0x14b6f38, 0xc00afc2000}, 0xc00f6f6a00, {0xc03992c208, 0x0, 0x1}, 0xa0?)
        /home/me/dev/nodes/XDPoSChain/consensus/XDPoS/engines/engine_v1/engine.go:216 +0x5a5
github.com/XinFinOrg/XDPoSChain/consensus/XDPoS/engines/engine_v1.(*XDPoS_v1).verifyHeaderWithCache(0xc00019d6b0, {0x14b6f38, 0xc00afc2000}, 0xc00ed6a570?, {0xc03992c208, 0x0, 0x1}, 0xc5?)
        /home/me/dev/nodes/XDPoSChain/consensus/XDPoS/engines/engine_v1/engine.go:143 +0x14b
github.com/XinFinOrg/XDPoSChain/consensus/XDPoS/engines/engine_v1.(*XDPoS_v1).VerifyHeaders.func1()
        /home/me/dev/nodes/XDPoSChain/consensus/XDPoS/engines/engine_v1/engine.go:126 +0x114
created by github.com/XinFinOrg/XDPoSChain/consensus/XDPoS/engines/engine_v1.(*XDPoS_v1).VerifyHeaders
        /home/me/dev/nodes/XDPoSChain/consensus/XDPoS/engines/engine_v1/engine.go:124 +0x16a

So now it crashes at block 59174100 validated by xdce865a5b2bf699a4b498de8a8c55da14bb0d94b21 despite the masternode set obtained at 59173650 containing that very validator (see in above logs).

@gzliudan
Copy link
Collaborator

Have you deleted old data before synchronize chain from snapshot ? What's your OS and golang version ?

@ircrp
Copy link
Author

ircrp commented May 24, 2023

Have you deleted old data before synchronize chain from snapshot ? What's your OS and golang version ?

Yes, all data was wiped out prior to synchronizing from the snapshot data. Following is the OS & golang version information

OS:    Ubuntu 22.04.1 LTS
GO:    go1.19 linux/amd64

@gzliudan
Copy link
Collaborator

Maybe the snapshot file should be updated. You can use golang v1.20.4 since there is panic error in your log.

@ircrp
Copy link
Author

ircrp commented May 24, 2023

Maybe the snapshot file should be updated. You can use golang v1.20.4 since there is panic error in your log.

Hi @gzliudan,

Thank you for your swift response and your suggestion to update my golang version. I appreciate your input and your expertise in this matter.

Before I proceed, I wanted to understand better. Could you please clarify how an update in the Golang version could potentially affect the validation issue that I'm currently experiencing? I ask because I have a colleague who is successfully running the node on go version go1.18.8 windows/amd64 without encountering this problem.

To my understanding, the current issue seems to be rooted in a masternode list mismatch at block 516600 and again at 59174100, which isn't inherently connected to the Golang version. I strongly believe that the primary issue is with the validation checks rather than a particular version of Golang.

However, if you can provide a solid explanation of how the Go version could influence this validation issue, I am more than willing to go ahead with your suggestion. I am keen to resolve this issue as swiftly as possible.

Thank you once again for your assistance and I look forward to your response.

@ircrp
Copy link
Author

ircrp commented May 24, 2023 •
edited
Loading

In addition to the above, I am curious if you were able to follow the reproduction steps I provided and not encounter the issue on your end. If so, could you please share any different steps you took or any specific modifications you made in your setup?

I'm particularly interested to see if updating the bootnodes list has resolved the issue for you. Understanding your process could be incredibly helpful in pinpointing where the discrepancy lies between our respective nodes.

@gzliudan
Copy link
Collaborator

I am researching how to update the signer list also. This issue is same with XinFinOrg/Local_DPoS_Setup#16.

The other problem in your second report is panic:

panic: runtime error: invalid memory address or nil pointer dereference

The panic is not related to synchronization. I guess it was caused by golang version. It is worth to test with another version of compiler.

I will get a new VM to test your problem tomorrow.

@gzliudan
Copy link
Collaborator

The go 1.19 has 9 minor revisions. Are you using v1.19.9 now ?

OS: Ubuntu 22.04.1 LTS
GO: go1.19 linux/amd64

@ircrp
Copy link
Author

ircrp commented May 25, 2023

Hi @gzliudan just tried latest go version this morning and still facing the same issue with go1.20.4. I've recorded the entire reproduction steps as a video and shared privately a youtube link to your @qq.com email in case that helps.

INFO [05-25|10:14:33] Updating new set of masternodes
INFO [05-25|10:14:33] New set of masternodes has been updated to snapshot number=516150 hash=25a20f…bc50cd new masternodes="[xdcFaa7A3c5b5a7f8c9A69f73347F9F28E2B9669d72 xdcc67c2DEc79da735d6587d8DB3c23271d557196ab xdc664c4a7B15d91B07c468162f535909114C038b91 xdcC7d49d0a2CF198DeEBd6ce581aF465944ec8b2bB xdc25c65b4b379ac37CF78357c4915f73677022eaFF xdcA0caeC455E7585d845BBF72420397774e52f3B74 xdced2cE9d056DB8DeF2B4d877c9a80b6f988ae202E xdc9a3787688fD210EC8F8D0224c6c50B8178d75Bc0 xdc72fb467Ef6dA285b6B0D8f7A25AbD6049385d5fD xdcA4F06BD6E4b217F1FfF9327BDE83da1d41193bE4 xdcDb2E141595d8Edf6B1CE40b10d57Ca5b83855ec5 xdc595b8170eAF2e53e47cc20db47aD063a1e6e2C0f xdc7923B7f78dFfacBb3e1b886e2410630975384C7F xdccfCCDeA1006a5CfA7d9484B5b293B46964c265c0 xdc7aA125338BE075260e77c6a66A56c90A5DeC4C58 xdc4398241671b3Dd484FE3213A4fB7511f30e7D7C0 xdca65010026b83368Ca05DF6E8B467985D6de3eAC5 xdc0eaa5c2F0929C95ED0280e8840A2416C4EF16E9d]"
INFO [05-25|10:14:33] Masternodes are ready for the next epoch
INFO [05-25|10:14:33] Imported new chain segment               blocks=450 txs=537 mgas=42.642 elapsed=273.000ms mgasps=156.198 number=516150 hash=25a20f…bc50cd cache=1.12mB
INFO [05-25|10:14:33] Persisted trie from memory database      nodes=1641 size=513.60kB time=9.844635ms   gcnodes=775 gcsize=277.97kB gctime=877.215µs  livenodes=2463 livesize=402.07kB
INFO [05-25|10:14:33] Persisted trie from memory database      nodes=1692 size=522.37kB time=6.191515ms   gcnodes=770 gcsize=273.66kB gctime=835.614µs  livenodes=2187 livesize=317.68kB
INFO [05-25|10:14:33] Persisted trie from memory database      nodes=1487 size=462.49kB time=5.53243ms    gcnodes=699 gcsize=249.23kB gctime=690.948µs  livenodes=2436 livesize=393.64kB
INFO [05-25|10:14:33] Imported new chain segment               blocks=449 txs=573 mgas=43.961 elapsed=245.386ms mgasps=179.151 number=516599 hash=a8731d…27b91a cache=1.40mB
ERROR[05-25|10:14:33] Masternodes lists are different in checkpoint header and snapshot number=516600 masternodes_from_checkpoint_header="[[6 85 81 240 220 172 111 0 202 225 17 146 212 98 219 112 155 227 117 140] [14 170 92 47 9 41 201 94 208 40 14 136 64 162 65 108 78 241 110 157] [37 198 91 75 55 154 195 124 247 131 87 196 145 95 115 103 112 34 234 255] [67 152 36 22 113 179 221 72 79 227 33 58 79 183 81 31 48 231 215 192] [89 91 129 112 234 242 229 62 71 204 32 219 71 173 6 58 30 110 44 15] [102 76 74 123 21 217 27 7 196 104 22 47 83 89 9 17 76 3 139 145] [114 251 70 126 246 218 40 91 107 13 143 122 37 171 214 4 147 133 213 253] [121 35 183 247 141 255 172 187 62 27 136 110 36 16 99 9 117 56 76 127] [122 161 37 51 139 224 117 38 14 119 198 166 106 86 201 10 93 236 76 88] [154 55 135 104 143 210 16 236 143 141 2 36 198 197 11 129 120 215 91 192] [160 202 236 69 94 117 133 216 69 187 247 36 32 57 119 116 229 47 59 116] [164 240 107 214 228 178 23 241 255 249 50 123 222 131 218 29 65 25 59 228] [199 212 157 10 44 241 152 222 235 214 206 88 26 244 101 148 78 200 178 187] [219 46 20 21 149 216 237 246 177 206 64 177 13 87 202 91 131 133 94 197] [166 80 16 2 107 131 54 140 160 93 246 232 180 103 152 93 109 227 234 197] [207 204 222 161 0 106 92 250 125 148 132 181 178 147 180 105 100 194 101 192] [237 44 233 208 86 219 141 239 43 77 135 124 154 128 182 249 136 174 32 46]]" masternodes_in_snapshot="[[14 170 92 47 9 41 201 94 208 40 14 136 64 162 65 108 78 241 110 157] [37 198 91 75 55 154 195 124 247 131 87 196 145 95 115 103 112 34 234 255] [67 152 36 22 113 179 221 72 79 227 33 58 79 183 81 31 48 231 215 192] [89 91 129 112 234 242 229 62 71 204 32 219 71 173 6 58 30 110 44 15] [102 76 74 123 21 217 27 7 196 104 22 47 83 89 9 17 76 3 139 145] [114 251 70 126 246 218 40 91 107 13 143 122 37 171 214 4 147 133 213 253] [121 35 183 247 141 255 172 187 62 27 136 110 36 16 99 9 117 56 76 127] [122 161 37 51 139 224 117 38 14 119 198 166 106 86 201 10 93 236 76 88] [154 55 135 104 143 210 16 236 143 141 2 36 198 197 11 129 120 215 91 192] [160 202 236 69 94 117 133 216 69 187 247 36 32 57 119 116 229 47 59 116] [164 240 107 214 228 178 23 241 255 249 50 123 222 131 218 29 65 25 59 228] [199 212 157 10 44 241 152 222 235 214 206 88 26 244 101 148 78 200 178 187] [219 46 20 21 149 216 237 246 177 206 64 177 13 87 202 91 131 133 94 197] [166 80 16 2 107 131 54 140 160 93 246 232 180 103 152 93 109 227 234 197] [198 124 45 236 121 218 115 93 101 135 216 219 60 35 39 29 85 113 150 171] [207 204 222 161 0 106 92 250 125 148 132 181 178 147 180 105 100 194 101 192] [237 44 233 208 86 219 141 239 43 77 135 124 154 128 182 249 136 174 32 46]]" penList=[]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x7822e4]

goroutine 10730739 [running]:
github.com/XinFinOrg/XDPoSChain/core/state.(*StateDB).getStateObject(0x0, {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...})
        /home/me/fresh_node/XDPoSChain/core/state/statedb.go:388 +0x64
github.com/XinFinOrg/XDPoSChain/core/state.(*StateDB).GetState(0x0, {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...)
        /home/me/fresh_node/XDPoSChain/core/state/statedb.go:256 +0x50
github.com/XinFinOrg/XDPoSChain/core/state.GetCandidates(0xf7a1ca23f336eb19?)
        /home/me/fresh_node/XDPoSChain/core/state/statedb_utils.go:95 +0x12c
github.com/XinFinOrg/XDPoSChain/eth/hooks.AttachConsensusV1Hooks.func5({0x25, 0xa2, 0xf, 0xa7, 0x58, 0x4f, 0x55, 0x59, 0xb5, 0xf0, ...})
        /home/me/fresh_node/XDPoSChain/eth/hooks/engine_v1_hooks.go:231 +0x16d
github.com/XinFinOrg/XDPoSChain/consensus/XDPoS/engines/engine_v1.(*XDPoS_v1).getSignersFromContract(0xc0004b47b0, {0x14c1378, 0xc00025c000}, 0xc0296ae500?)
        /home/me/fresh_node/XDPoSChain/consensus/XDPoS/engines/engine_v1/engine.go:1003 +0x158
github.com/XinFinOrg/XDPoSChain/consensus/XDPoS/engines/engine_v1.(*XDPoS_v1).verifyCascadingFields(0xc0004b47b0, {0x14c1378, 0xc00025c000}, 0xc0296ae500, {0xc027a1f7e8?, 0x0, 0x1}, 0x1c?)
        /home/me/fresh_node/XDPoSChain/consensus/XDPoS/engines/engine_v1/engine.go:263 +0x345
github.com/XinFinOrg/XDPoSChain/consensus/XDPoS/engines/engine_v1.(*XDPoS_v1).verifyHeader(0xc0004b47b0, {0x14c1378, 0xc00025c000}, 0xc0296ae500, {0xc027a1f7e8, 0x0, 0x1}, 0x80?)
        /home/me/fresh_node/XDPoSChain/consensus/XDPoS/engines/engine_v1/engine.go:216 +0x605
github.com/XinFinOrg/XDPoSChain/consensus/XDPoS/engines/engine_v1.(*XDPoS_v1).verifyHeaderWithCache(0xc0004b47b0, {0x14c1378, 0xc00025c000}, 0xc044f11720?, {0xc027a1f7e8, 0x0, 0x1}, 0x1?)
        /home/me/fresh_node/XDPoSChain/consensus/XDPoS/engines/engine_v1/engine.go:143 +0x12b
github.com/XinFinOrg/XDPoSChain/consensus/XDPoS/engines/engine_v1.(*XDPoS_v1).VerifyHeaders.func1()
        /home/me/fresh_node/XDPoSChain/consensus/XDPoS/engines/engine_v1/engine.go:126 +0x114
created by github.com/XinFinOrg/XDPoSChain/consensus/XDPoS/engines/engine_v1.(*XDPoS_v1).VerifyHeaders
        /home/me/fresh_node/XDPoSChain/consensus/XDPoS/engines/engine_v1/engine.go:124 +0x16a
me@me-NPx5:~/fresh_node/XDPoSChain$ go version
go version go1.20.4 linux/amd64

@gzliudan
Copy link
Collaborator

I'm testing your problems now.

@gzliudan
Copy link
Collaborator

Sorry, I didn't receive your e-mail. I checked my mail box [email protected] just now.

@ircrp
Copy link
Author

ircrp commented May 25, 2023

[email protected]

I guess the youtube sharing didn't work, just sent you an email with the link. Hopefully you get it now. Cheers !

@gzliudan
Copy link
Collaborator

Received your email now. According to your steps in first report, the error messages are as below. There is a panic message already.

INFO [05-25|06:07:50] Checkpoint!!! It's time to reconcile node's state... 
INFO [05-25|06:07:50] Persisted trie from memory database      nodes=1476 size=462.24kB time=13.564822ms gcnodes=674 gcsize=240.38kB gctime=1.437413ms livenodes=2470 livesize=410.75kB
INFO [05-25|06:07:50] Persisted trie from memory database      nodes=1665 size=512.85kB time=13.605823ms gcnodes=804 gcsize=284.31kB gctime=1.822217ms livenodes=2141 livesize=322.01kB
INFO [05-25|06:07:51] Persisted trie from memory database      nodes=1592 size=498.99kB time=10.576695ms gcnodes=779 gcsize=283.09kB gctime=1.820115ms livenodes=2112 livesize=294.78kB
INFO [05-25|06:07:51] Persisted trie from memory database      nodes=1479 size=459.89kB time=10.152191ms gcnodes=632 gcsize=221.47kB gctime=1.35551ms  livenodes=2329 livesize=378.45kB
INFO [05-25|06:07:51] It's time to update new set of masternodes for the next epoch... 
INFO [05-25|06:07:51] Ordered list of masternode candidates 
INFO [05-25|06:07:51]                                          address=xdcFaa7A3c5b5a7f8c9A69f73347F9F28E2B9669d72 stake=10000000000000000000000000
INFO [05-25|06:07:51]                                          address=xdcA4F06BD6E4b217F1FfF9327BDE83da1d41193bE4 stake=10000000000000000000000000
INFO [05-25|06:07:51]                                          address=xdc664c4a7B15d91B07c468162f535909114C038b91 stake=10000000000000000000000000
INFO [05-25|06:07:51]                                          address=xdca65010026b83368Ca05DF6E8B467985D6de3eAC5 stake=10000000000000000000000000
INFO [05-25|06:07:51]                                          address=xdc0eaa5c2F0929C95ED0280e8840A2416C4EF16E9d stake=10000000000000000000000000
INFO [05-25|06:07:51]                                          address=xdcA0caeC455E7585d845BBF72420397774e52f3B74 stake=10000000000000000000000000
INFO [05-25|06:07:51]                                          address=xdced2cE9d056DB8DeF2B4d877c9a80b6f988ae202E stake=10000000000000000000000000
INFO [05-25|06:07:51]                                          address=xdc9a3787688fD210EC8F8D0224c6c50B8178d75Bc0 stake=10000000000000000000000000
INFO [05-25|06:07:51]                                          address=xdc25c65b4b379ac37CF78357c4915f73677022eaFF stake=10000000000000000000000000
INFO [05-25|06:07:51]                                          address=xdcc67c2DEc79da735d6587d8DB3c23271d557196ab stake=10000000000000000000000000
INFO [05-25|06:07:51]                                          address=xdcDb2E141595d8Edf6B1CE40b10d57Ca5b83855ec5 stake=10000000000000000000000000
INFO [05-25|06:07:51]                                          address=xdc595b8170eAF2e53e47cc20db47aD063a1e6e2C0f stake=10000000000000000000000000
INFO [05-25|06:07:51]                                          address=xdc7923B7f78dFfacBb3e1b886e2410630975384C7F stake=10000000000000000000000000
INFO [05-25|06:07:51]                                          address=xdccfCCDeA1006a5CfA7d9484B5b293B46964c265c0 stake=10000000000000000000000000
INFO [05-25|06:07:51]                                          address=xdc7aA125338BE075260e77c6a66A56c90A5DeC4C58 stake=10000000000000000000000000
INFO [05-25|06:07:51]                                          address=xdc4398241671b3Dd484FE3213A4fB7511f30e7D7C0 stake=10000000000000000000000000
INFO [05-25|06:07:51]                                          address=xdcC7d49d0a2CF198DeEBd6ce581aF465944ec8b2bB stake=10000000000000000000000000
INFO [05-25|06:07:51]                                          address=xdc065551F0dcAC6f00CAe11192D462db709bE3758c stake=10000000000000000000000000
INFO [05-25|06:07:51] Updating new set of masternodes 
INFO [05-25|06:07:51] New set of masternodes has been updated to snapshot number=514350 hash=ae1ee9…2a680d new masternodes="[xdcFaa7A3c5b5a7f8c9A69f73347F9F28E2B9669d72 xdcA4F06BD6E4b217F1FfF9327BDE83da1d41193bE4 xdc664c4a7B15d91B07c468162f535909114C038b91 xdca65010026b83368Ca05DF6E8B467985D6de3eAC5 xdc0eaa5c2F0929C95ED0280e8840A2416C4EF16E9d xdcA0caeC455E7585d845BBF72420397774e52f3B74 xdced2cE9d056DB8DeF2B4d877c9a80b6f988ae202E xdc9a3787688fD210EC8F8D0224c6c50B8178d75Bc0 xdc25c65b4b379ac37CF78357c4915f73677022eaFF xdcc67c2DEc79da735d6587d8DB3c23271d557196ab xdcDb2E141595d8Edf6B1CE40b10d57Ca5b83855ec5 xdc595b8170eAF2e53e47cc20db47aD063a1e6e2C0f xdc7923B7f78dFfacBb3e1b886e2410630975384C7F xdccfCCDeA1006a5CfA7d9484B5b293B46964c265c0 xdc7aA125338BE075260e77c6a66A56c90A5DeC4C58 xdc4398241671b3Dd484FE3213A4fB7511f30e7D7C0 xdcC7d49d0a2CF198DeEBd6ce581aF465944ec8b2bB xdc065551F0dcAC6f00CAe11192D462db709bE3758c]"
INFO [05-25|06:07:51] Masternodes are ready for the next epoch 
INFO [05-25|06:07:51] Imported new chain segment               blocks=450 txs=534 mgas=42.380 elapsed=750.877ms mgasps=56.440  number=514350 hash=ae1ee9…2a680d cache=1.07mB
INFO [05-25|06:07:51] Persisted trie from memory database      nodes=1602 size=508.35kB time=11.862907ms gcnodes=726 gcsize=256.75kB gctime=1.641116ms livenodes=2083 livesize=294.22kB
INFO [05-25|06:07:51] Persisted trie from memory database      nodes=1445 size=456.30kB time=10.559295ms gcnodes=637 gcsize=224.66kB gctime=1.40051ms  livenodes=2363 livesize=379.12kB
INFO [05-25|06:07:52] Persisted trie from memory database      nodes=1633 size=506.52kB time=12.530413ms gcnodes=729 gcsize=259.42kB gctime=1.661419ms livenodes=2193 livesize=332.62kB
INFO [05-25|06:07:52] Imported new chain segment               blocks=449 txs=570 mgas=43.462 elapsed=878.457ms mgasps=49.475  number=514799 hash=6d998f…4977fd cache=1.40mB
INFO [05-25|06:07:52] Calculate reward at checkpoint           startBlock=513001 endBlock=513900
INFO [05-25|06:07:52] Signers data                             signers="{\"xdc065551f0dcac6f00cae11192d462db709be3758c\":{\"sign\":60,\"reward\":297324083250743310180},\"xdc0eaa5c2f0929c95ed0280e8840a2416c4ef16e9d\":{\"sign\":49,\"reward\":242814667988107036647},\"xdc25c65b4b379ac37cf78357c4915f73677022eaff\":{\"sign\":60,\"reward\":297324083250743310180},\"xdc4398241671b3dd484fe3213a4fb7511f30e7d7c0\":{\"sign\":60,\"reward\":297324083250743310180},\"xdc595b8170eaf2e53e47cc20db47ad063a1e6e2c0f\":{\"sign\":60,\"reward\":297324083250743310180},\"xdc664c4a7b15d91b07c468162f535909114c038b91\":{\"sign\":60,\"reward\":297324083250743310180},\"xdc7923b7f78dffacbb3e1b886e2410630975384c7f\":{\"sign\":60,\"reward\":297324083250743310180},\"xdc7aa125338be075260e77c6a66a56c90a5dec4c58\":{\"sign\":60,\"reward\":297324083250743310180},\"xdc9a3787688fd210ec8f8d0224c6c50b8178d75bc0\":{\"sign\":60,\"reward\":297324083250743310180},\"xdca0caec455e7585d845bbf72420397774e52f3b74\":{\"sign\":60,\"reward\":297324083250743310180},\"xdca4f06bd6e4b217f1fff9327bde83da1d41193be4\":{\"sign\":60,\"reward\":297324083250743310180},\"xdca65010026b83368ca05df6e8b467985d6de3eac5\":{\"sign\":60,\"reward\":297324083250743310180},\"xdcc67c2dec79da735d6587d8db3c23271d557196ab\":{\"sign\":60,\"reward\":297324083250743310180},\"xdcc7d49d0a2cf198deebd6ce581af465944ec8b2bb\":{\"sign\":60,\"reward\":297324083250743310180},\"xdccfccdea1006a5cfa7d9484b5b293b46964c265c0\":{\"sign\":60,\"reward\":297324083250743310180},\"xdcdb2e141595d8edf6b1ce40b10d57ca5b83855ec5\":{\"sign\":60,\"reward\":297324083250743310180},\"xdced2ce9d056db8def2b4d877c9a80b6f988ae202e\":{\"sign\":60,\"reward\":297324083250743310180}}"                                                                                                totalSigner=1009 totalReward=5000000000000000000000
INFO [05-25|06:07:52] Imported new chain segment               blocks=1   txs=0   mgas=0.000  elapsed=68.890ms  mgasps=0.000   number=514800 hash=cb233a…7bdc85 cache=1.41mB
INFO [05-25|06:07:52] Checkpoint!!! It's time to reconcile node's state... 
INFO [05-25|06:07:52] Persisted trie from memory database      nodes=1642 size=509.50kB time=14.46123ms  gcnodes=781 gcsize=281.79kB gctime=1.929621ms livenodes=2191 livesize=321.62kB
INFO [05-25|06:07:52] Persisted trie from memory database      nodes=1498 size=461.77kB time=13.353021ms gcnodes=692 gcsize=247.00kB gctime=1.829518ms livenodes=2380 livesize=388.69kB
INFO [05-25|06:07:52] Persisted trie from memory database      nodes=1606 size=500.49kB time=12.484612ms gcnodes=773 gcsize=275.43kB gctime=1.923121ms livenodes=2144 livesize=315.06kB
INFO [05-25|06:07:53] Persisted trie from memory database      nodes=1477 size=467.18kB time=13.851625ms gcnodes=666 gcsize=235.19kB gctime=1.363615ms livenodes=2383 livesize=398.67kB
INFO [05-25|06:07:53] It's time to update new set of masternodes for the next epoch... 
INFO [05-25|06:07:53] Ordered list of masternode candidates 
INFO [05-25|06:07:53]                                          address=xdcFaa7A3c5b5a7f8c9A69f73347F9F28E2B9669d72 stake=10000000000000000000000000
INFO [05-25|06:07:53]                                          address=xdcA4F06BD6E4b217F1FfF9327BDE83da1d41193bE4 stake=10000000000000000000000000
INFO [05-25|06:07:53]                                          address=xdc664c4a7B15d91B07c468162f535909114C038b91 stake=10000000000000000000000000
INFO [05-25|06:07:53]                                          address=xdca65010026b83368Ca05DF6E8B467985D6de3eAC5 stake=10000000000000000000000000
INFO [05-25|06:07:53]                                          address=xdc0eaa5c2F0929C95ED0280e8840A2416C4EF16E9d stake=10000000000000000000000000
INFO [05-25|06:07:53]                                          address=xdcA0caeC455E7585d845BBF72420397774e52f3B74 stake=10000000000000000000000000
INFO [05-25|06:07:53]                                          address=xdced2cE9d056DB8DeF2B4d877c9a80b6f988ae202E stake=10000000000000000000000000
INFO [05-25|06:07:53]                                          address=xdc9a3787688fD210EC8F8D0224c6c50B8178d75Bc0 stake=10000000000000000000000000
INFO [05-25|06:07:53]                                          address=xdc25c65b4b379ac37CF78357c4915f73677022eaFF stake=10000000000000000000000000
INFO [05-25|06:07:53]                                          address=xdcc67c2DEc79da735d6587d8DB3c23271d557196ab stake=10000000000000000000000000
INFO [05-25|06:07:53]                                          address=xdcDb2E141595d8Edf6B1CE40b10d57Ca5b83855ec5 stake=10000000000000000000000000
INFO [05-25|06:07:53]                                          address=xdc595b8170eAF2e53e47cc20db47aD063a1e6e2C0f stake=10000000000000000000000000
INFO [05-25|06:07:53]                                          address=xdc7923B7f78dFfacBb3e1b886e2410630975384C7F stake=10000000000000000000000000
INFO [05-25|06:07:53]                                          address=xdccfCCDeA1006a5CfA7d9484B5b293B46964c265c0 stake=10000000000000000000000000
INFO [05-25|06:07:53]                                          address=xdc7aA125338BE075260e77c6a66A56c90A5DeC4C58 stake=10000000000000000000000000
INFO [05-25|06:07:53]                                          address=xdc4398241671b3Dd484FE3213A4fB7511f30e7D7C0 stake=10000000000000000000000000
INFO [05-25|06:07:53]                                          address=xdcC7d49d0a2CF198DeEBd6ce581aF465944ec8b2bB stake=10000000000000000000000000
INFO [05-25|06:07:53]                                          address=xdc065551F0dcAC6f00CAe11192D462db709bE3758c stake=10000000000000000000000000
INFO [05-25|06:07:53] Updating new set of masternodes 
INFO [05-25|06:07:53] New set of masternodes has been updated to snapshot number=515250 hash=67be97…c86ee0 new masternodes="[xdcFaa7A3c5b5a7f8c9A69f73347F9F28E2B9669d72 xdcA4F06BD6E4b217F1FfF9327BDE83da1d41193bE4 xdc664c4a7B15d91B07c468162f535909114C038b91 xdca65010026b83368Ca05DF6E8B467985D6de3eAC5 xdc0eaa5c2F0929C95ED0280e8840A2416C4EF16E9d xdcA0caeC455E7585d845BBF72420397774e52f3B74 xdced2cE9d056DB8DeF2B4d877c9a80b6f988ae202E xdc9a3787688fD210EC8F8D0224c6c50B8178d75Bc0 xdc25c65b4b379ac37CF78357c4915f73677022eaFF xdcc67c2DEc79da735d6587d8DB3c23271d557196ab xdcDb2E141595d8Edf6B1CE40b10d57Ca5b83855ec5 xdc595b8170eAF2e53e47cc20db47aD063a1e6e2C0f xdc7923B7f78dFfacBb3e1b886e2410630975384C7F xdccfCCDeA1006a5CfA7d9484B5b293B46964c265c0 xdc7aA125338BE075260e77c6a66A56c90A5DeC4C58 xdc4398241671b3Dd484FE3213A4fB7511f30e7D7C0 xdcC7d49d0a2CF198DeEBd6ce581aF465944ec8b2bB xdc065551F0dcAC6f00CAe11192D462db709bE3758c]"
INFO [05-25|06:07:53] Masternodes are ready for the next epoch 
INFO [05-25|06:07:53] Imported new chain segment               blocks=450 txs=533 mgas=42.303 elapsed=746.384ms mgasps=56.678  number=515250 hash=67be97…c86ee0 cache=1.09mB
INFO [05-25|06:07:53] Persisted trie from memory database      nodes=1615 size=509.31kB time=16.914452ms gcnodes=767 gcsize=276.75kB gctime=1.656613ms livenodes=2180 livesize=321.02kB
INFO [05-25|06:07:53] Persisted trie from memory database      nodes=1641 size=509.71kB time=11.155101ms gcnodes=751 gcsize=268.76kB gctime=1.713315ms livenodes=2106 livesize=310.08kB
INFO [05-25|06:07:53] Persisted trie from memory database      nodes=1441 size=459.92kB time=11.491404ms gcnodes=664 gcsize=237.72kB gctime=1.396713ms livenodes=2507 livesize=422.90kB
INFO [05-25|06:07:53] Imported new chain segment               blocks=394 txs=503 mgas=39.299 elapsed=651.811ms mgasps=60.292  number=515644 hash=ca0968…a9e335 cache=1.22mB
INFO [05-25|06:07:53] Imported new chain segment               blocks=55  txs=72  mgas=4.803  elapsed=87.725ms  mgasps=54.748  number=515699 hash=7f3861…271ec6 cache=1.41mB
INFO [05-25|06:07:53] Calculate reward at checkpoint           startBlock=513901 endBlock=514800
INFO [05-25|06:07:53] Signers data                             signers="{\"xdc065551f0dcac6f00cae11192d462db709be3758c\":{\"sign\":60,\"reward\":297914597815292949300},\"xdc0eaa5c2f0929c95ed0280e8840a2416c4ef16e9d\":{\"sign\":47,\"reward\":233366434955312810285},\"xdc25c65b4b379ac37cf78357c4915f73677022eaff\":{\"sign\":60,\"reward\":297914597815292949300},\"xdc4398241671b3dd484fe3213a4fb7511f30e7d7c0\":{\"sign\":60,\"reward\":297914597815292949300},\"xdc595b8170eaf2e53e47cc20db47ad063a1e6e2c0f\":{\"sign\":60,\"reward\":297914597815292949300},\"xdc664c4a7b15d91b07c468162f535909114c038b91\":{\"sign\":60,\"reward\":297914597815292949300},\"xdc7923b7f78dffacbb3e1b886e2410630975384c7f\":{\"sign\":60,\"reward\":297914597815292949300},\"xdc7aa125338be075260e77c6a66a56c90a5dec4c58\":{\"sign\":60,\"reward\":297914597815292949300},\"xdc9a3787688fd210ec8f8d0224c6c50b8178d75bc0\":{\"sign\":60,\"reward\":297914597815292949300},\"xdca0caec455e7585d845bbf72420397774e52f3b74\":{\"sign\":60,\"reward\":297914597815292949300},\"xdca4f06bd6e4b217f1fff9327bde83da1d41193be4\":{\"sign\":60,\"reward\":297914597815292949300},\"xdca65010026b83368ca05df6e8b467985d6de3eac5\":{\"sign\":60,\"reward\":297914597815292949300},\"xdcc67c2dec79da735d6587d8db3c23271d557196ab\":{\"sign\":60,\"reward\":297914597815292949300},\"xdcc7d49d0a2cf198deebd6ce581af465944ec8b2bb\":{\"sign\":60,\"reward\":297914597815292949300},\"xdccfccdea1006a5cfa7d9484b5b293b46964c265c0\":{\"sign\":60,\"reward\":297914597815292949300},\"xdcdb2e141595d8edf6b1ce40b10d57ca5b83855ec5\":{\"sign\":60,\"reward\":297914597815292949300},\"xdced2ce9d056db8def2b4d877c9a80b6f988ae202e\":{\"sign\":60,\"reward\":297914597815292949300}}"                                                                                                totalSigner=1007 totalReward=5000000000000000000000
INFO [05-25|06:07:53] Imported new chain segment               blocks=1   txs=0   mgas=0.000  elapsed=71.611ms  mgasps=0.000   number=515700 hash=b239c2…075c66 cache=1.42mB
INFO [05-25|06:07:53] Checkpoint!!! It's time to reconcile node's state... 
INFO [05-25|06:07:53] Persisted trie from memory database      nodes=1703 size=520.42kB time=21.516994ms gcnodes=803 gcsize=290.28kB gctime=1.986315ms livenodes=2177 livesize=322.02kB
INFO [05-25|06:07:54] Persisted trie from memory database      nodes=1508 size=471.65kB time=15.824743ms gcnodes=668 gcsize=238.33kB gctime=1.787614ms livenodes=2371 livesize=385.80kB
INFO [05-25|06:07:54] Persisted trie from memory database      nodes=1608 size=499.91kB time=11.231202ms gcnodes=762 gcsize=273.94kB gctime=1.832517ms livenodes=2376 livesize=388.38kB
INFO [05-25|06:07:54] Persisted trie from memory database      nodes=1621 size=506.84kB time=12.526113ms gcnodes=754 gcsize=269.68kB gctime=1.783117ms livenodes=2168 livesize=323.25kB
INFO [05-25|06:07:54] It's time to update new set of masternodes for the next epoch... 
INFO [05-25|06:07:54] Ordered list of masternode candidates 
INFO [05-25|06:07:54]                                          address=xdcFaa7A3c5b5a7f8c9A69f73347F9F28E2B9669d72 stake=10000000000000000000000000
INFO [05-25|06:07:54]                                          address=xdcc67c2DEc79da735d6587d8DB3c23271d557196ab stake=10000000000000000000000000
INFO [05-25|06:07:54]                                          address=xdc664c4a7B15d91B07c468162f535909114C038b91 stake=10000000000000000000000000
INFO [05-25|06:07:54]                                          address=xdcC7d49d0a2CF198DeEBd6ce581aF465944ec8b2bB stake=10000000000000000000000000
INFO [05-25|06:07:54]                                          address=xdc25c65b4b379ac37CF78357c4915f73677022eaFF stake=10000000000000000000000000
INFO [05-25|06:07:54]                                          address=xdcA0caeC455E7585d845BBF72420397774e52f3B74 stake=10000000000000000000000000
INFO [05-25|06:07:54]                                          address=xdced2cE9d056DB8DeF2B4d877c9a80b6f988ae202E stake=10000000000000000000000000
INFO [05-25|06:07:54]                                          address=xdc9a3787688fD210EC8F8D0224c6c50B8178d75Bc0 stake=10000000000000000000000000
INFO [05-25|06:07:54]                                          address=xdc72fb467Ef6dA285b6B0D8f7A25AbD6049385d5fD stake=10000000000000000000000000
INFO [05-25|06:07:54]                                          address=xdcA4F06BD6E4b217F1FfF9327BDE83da1d41193bE4 stake=10000000000000000000000000
INFO [05-25|06:07:54]                                          address=xdcDb2E141595d8Edf6B1CE40b10d57Ca5b83855ec5 stake=10000000000000000000000000
INFO [05-25|06:07:54]                                          address=xdc595b8170eAF2e53e47cc20db47aD063a1e6e2C0f stake=10000000000000000000000000
INFO [05-25|06:07:54]                                          address=xdc7923B7f78dFfacBb3e1b886e2410630975384C7F stake=10000000000000000000000000
INFO [05-25|06:07:54]                                          address=xdccfCCDeA1006a5CfA7d9484B5b293B46964c265c0 stake=10000000000000000000000000
INFO [05-25|06:07:54]                                          address=xdc7aA125338BE075260e77c6a66A56c90A5DeC4C58 stake=10000000000000000000000000
INFO [05-25|06:07:54]                                          address=xdc4398241671b3Dd484FE3213A4fB7511f30e7D7C0 stake=10000000000000000000000000
INFO [05-25|06:07:54]                                          address=xdca65010026b83368Ca05DF6E8B467985D6de3eAC5 stake=10000000000000000000000000
INFO [05-25|06:07:54]                                          address=xdc0eaa5c2F0929C95ED0280e8840A2416C4EF16E9d stake=10000000000000000000000000
INFO [05-25|06:07:54]                                          address=xdc065551F0dcAC6f00CAe11192D462db709bE3758c stake=10000000000000000000000000
INFO [05-25|06:07:54] Updating new set of masternodes 
INFO [05-25|06:07:54] New set of masternodes has been updated to snapshot number=516150 hash=25a20f…bc50cd new masternodes="[xdcFaa7A3c5b5a7f8c9A69f73347F9F28E2B9669d72 xdcc67c2DEc79da735d6587d8DB3c23271d557196ab xdc664c4a7B15d91B07c468162f535909114C038b91 xdcC7d49d0a2CF198DeEBd6ce581aF465944ec8b2bB xdc25c65b4b379ac37CF78357c4915f73677022eaFF xdcA0caeC455E7585d845BBF72420397774e52f3B74 xdced2cE9d056DB8DeF2B4d877c9a80b6f988ae202E xdc9a3787688fD210EC8F8D0224c6c50B8178d75Bc0 xdc72fb467Ef6dA285b6B0D8f7A25AbD6049385d5fD xdcA4F06BD6E4b217F1FfF9327BDE83da1d41193bE4 xdcDb2E141595d8Edf6B1CE40b10d57Ca5b83855ec5 xdc595b8170eAF2e53e47cc20db47aD063a1e6e2C0f xdc7923B7f78dFfacBb3e1b886e2410630975384C7F xdccfCCDeA1006a5CfA7d9484B5b293B46964c265c0 xdc7aA125338BE075260e77c6a66A56c90A5DeC4C58 xdc4398241671b3Dd484FE3213A4fB7511f30e7D7C0 xdca65010026b83368Ca05DF6E8B467985D6de3eAC5 xdc0eaa5c2F0929C95ED0280e8840A2416C4EF16E9d]"
INFO [05-25|06:07:54] Masternodes are ready for the next epoch 
INFO [05-25|06:07:54] Imported new chain segment               blocks=450 txs=537 mgas=42.642 elapsed=769.906ms mgasps=55.386  number=516150 hash=25a20f…bc50cd cache=1.03mB
INFO [05-25|06:07:54] Persisted trie from memory database      nodes=1476 size=465.02kB time=14.668533ms gcnodes=691 gcsize=246.44kB gctime=1.514012ms livenodes=2463 livesize=407.65kB
INFO [05-25|06:07:55] Persisted trie from memory database      nodes=1692 size=522.37kB time=12.382711ms gcnodes=770 gcsize=273.66kB gctime=1.975318ms livenodes=2187 livesize=323.26kB
INFO [05-25|06:07:55] Persisted trie from memory database      nodes=1487 size=462.49kB time=13.157319ms gcnodes=699 gcsize=249.23kB gctime=1.568515ms livenodes=2436 livesize=399.22kB
INFO [05-25|06:07:55] Imported new chain segment               blocks=449 txs=573 mgas=43.961 elapsed=746.348ms mgasps=58.902  number=516599 hash=a8731d…27b91a cache=1.40mB
ERROR[05-25|06:07:55] Masternodes lists are different in checkpoint header and snapshot number=516600 masternodes_from_checkpoint_header="[[6 85 81 240 220 172 111 0 202 225 17 146 212 98 219 112 155 227 117 140] [14 170 92 47 9 41 201 94 208 40 14 136 64 162 65 108 78 241 110 157] [37 198 91 75 55 154 195 124 247 131 87 196 145 95 115 103 112 34 234 255] [67 152 36 22 113 179 221 72 79 227 33 58 79 183 81 31 48 231 215 192] [89 91 129 112 234 242 229 62 71 204 32 219 71 173 6 58 30 110 44 15] [102 76 74 123 21 217 27 7 196 104 22 47 83 89 9 17 76 3 139 145] [114 251 70 126 246 218 40 91 107 13 143 122 37 171 214 4 147 133 213 253] [121 35 183 247 141 255 172 187 62 27 136 110 36 16 99 9 117 56 76 127] [122 161 37 51 139 224 117 38 14 119 198 166 106 86 201 10 93 236 76 88] [154 55 135 104 143 210 16 236 143 141 2 36 198 197 11 129 120 215 91 192] [160 202 236 69 94 117 133 216 69 187 247 36 32 57 119 116 229 47 59 116] [164 240 107 214 228 178 23 241 255 249 50 123 222 131 218 29 65 25 59 228] [199 212 157 10 44 241 152 222 235 214 206 88 26 244 101 148 78 200 178 187] [219 46 20 21 149 216 237 246 177 206 64 177 13 87 202 91 131 133 94 197] [166 80 16 2 107 131 54 140 160 93 246 232 180 103 152 93 109 227 234 197] [207 204 222 161 0 106 92 250 125 148 132 181 178 147 180 105 100 194 101 192] [237 44 233 208 86 219 141 239 43 77 135 124 154 128 182 249 136 174 32 46]]" masternodes_in_snapshot="[[14 170 92 47 9 41 201 94 208 40 14 136 64 162 65 108 78 241 110 157] [37 198 91 75 55 154 195 124 247 131 87 196 145 95 115 103 112 34 234 255] [67 152 36 22 113 179 221 72 79 227 33 58 79 183 81 31 48 231 215 192] [89 91 129 112 234 242 229 62 71 204 32 219 71 173 6 58 30 110 44 15] [102 76 74 123 21 217 27 7 196 104 22 47 83 89 9 17 76 3 139 145] [114 251 70 126 246 218 40 91 107 13 143 122 37 171 214 4 147 133 213 253] [121 35 183 247 141 255 172 187 62 27 136 110 36 16 99 9 117 56 76 127] [122 161 37 51 139 224 117 38 14 119 198 166 106 86 201 10 93 236 76 88] [154 55 135 104 143 210 16 236 143 141 2 36 198 197 11 129 120 215 91 192] [160 202 236 69 94 117 133 216 69 187 247 36 32 57 119 116 229 47 59 116] [164 240 107 214 228 178 23 241 255 249 50 123 222 131 218 29 65 25 59 228] [199 212 157 10 44 241 152 222 235 214 206 88 26 244 101 148 78 200 178 187] [219 46 20 21 149 216 237 246 177 206 64 177 13 87 202 91 131 133 94 197] [166 80 16 2 107 131 54 140 160 93 246 232 180 103 152 93 109 227 234 197] [198 124 45 236 121 218 115 93 101 135 216 219 60 35 39 29 85 113 150 171] [207 204 222 161 0 106 92 250 125 148 132 181 178 147 180 105 100 194 101 192] [237 44 233 208 86 219 141 239 43 77 135 124 154 128 182 249 136 174 32 46]]" penList=[]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x7822e4]

goroutine 4538456 [running]:
github.com/XinFinOrg/XDPoSChain/core/state.(*StateDB).getStateObject(0x0, {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...})
	/home/liudan/XDPoSChain/core/state/statedb.go:388 +0x64
github.com/XinFinOrg/XDPoSChain/core/state.(*StateDB).GetState(0x0, {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...)
	/home/liudan/XDPoSChain/core/state/statedb.go:256 +0x50
github.com/XinFinOrg/XDPoSChain/core/state.GetCandidates(0xf7a1ca23f336eb19?)
	/home/liudan/XDPoSChain/core/state/statedb_utils.go:95 +0x12c
github.com/XinFinOrg/XDPoSChain/eth/hooks.AttachConsensusV1Hooks.func5({0x25, 0xa2, 0xf, 0xa7, 0x58, 0x4f, 0x55, 0x59, 0xb5, 0xf0, ...})
	/home/liudan/XDPoSChain/eth/hooks/engine_v1_hooks.go:231 +0x16d
github.com/XinFinOrg/XDPoSChain/consensus/XDPoS/engines/engine_v1.(*XDPoS_v1).getSignersFromContract(0xc0001f9110, {0x14c1378, 0xc0002f8000}, 0xc03b968500?)
	/home/liudan/XDPoSChain/consensus/XDPoS/engines/engine_v1/engine.go:1003 +0x158
github.com/XinFinOrg/XDPoSChain/consensus/XDPoS/engines/engine_v1.(*XDPoS_v1).verifyCascadingFields(0xc0001f9110, {0x14c1378, 0xc0002f8000}, 0xc03b968500, {0xc02297f2f0?, 0x0, 0x1}, 0x1c?)
	/home/liudan/XDPoSChain/consensus/XDPoS/engines/engine_v1/engine.go:263 +0x345
github.com/XinFinOrg/XDPoSChain/consensus/XDPoS/engines/engine_v1.(*XDPoS_v1).verifyHeader(0xc0001f9110, {0x14c1378, 0xc0002f8000}, 0xc03b968500, {0xc02297f2f0, 0x0, 0x1}, 0x40?)
	/home/liudan/XDPoSChain/consensus/XDPoS/engines/engine_v1/engine.go:216 +0x605
github.com/XinFinOrg/XDPoSChain/consensus/XDPoS/engines/engine_v1.(*XDPoS_v1).verifyHeaderWithCache(0xc0001f9110, {0x14c1378, 0xc0002f8000}, 0x6d7bc7?, {0xc02297f2f0, 0x0, 0x1}, 0x65?)
	/home/liudan/XDPoSChain/consensus/XDPoS/engines/engine_v1/engine.go:143 +0x12b
github.com/XinFinOrg/XDPoSChain/consensus/XDPoS/engines/engine_v1.(*XDPoS_v1).VerifyHeaders.func1()
	/home/liudan/XDPoSChain/consensus/XDPoS/engines/engine_v1/engine.go:126 +0x114
created by github.com/XinFinOrg/XDPoSChain/consensus/XDPoS/engines/engine_v1.(*XDPoS_v1).VerifyHeaders
	/home/liudan/XDPoSChain/consensus/XDPoS/engines/engine_v1/engine.go:124 +0x16a

Steps to reproduce:

git clone https://github.com/XinFinOrg/XDPoSChain.git
git checkout v1.4.6
wget https://raw.githubusercontent.com/XinFinOrg/XinFin-Node/master/mainnet/genesis.json
make XDC
./build/bin/XDC --datadir /home/me/.xinfin init genesis.json
./build/bin/XDC --datadir /home/me/.xinfin --networkid 50 --etherbase 0x000000000000000000000000000000000000dead

Error:

INFO [05-20|09:42:55] Imported new chain segment               blocks=449 txs=573 mgas=43.961 elapsed=259.648ms mgasps=169.311 number=516599 hash=a8731d…27b91a cache=1.40mB
ERROR[05-20|09:42:55] Masternodes lists are different in checkpoint header and snapshot number=516600 masternodes_from_checkpoint_header="...."

@ircrp
Copy link
Author

ircrp commented May 25, 2023

Hi @gzliudan,

From your recent logs, it's clear that you've managed to successfully reproduce the issue. This is reassuring on one hand, as it confirms that it's not an isolated problem. However, on the other hand, it's quite concerning given that this is preventing individuals from initiating nodes, which in turn could significantly impact the network's operation.

In light of these findings, could I kindly request you to escalate this matter within your team? This will hopefully ensure that it receives the necessary priority for swift resolution. Many of us are keenly awaiting the opportunity to start nodes and contribute to the vitality of the network.

Thank you for your unwavering commitment and support in addressing these issues.

@gzliudan
Copy link
Collaborator

According to my debug work today, this line https://github.com/XinFinOrg/XDPoSChain/blob/master/consensus/XDPoS/engines/engine_v1/engine.go#L263 cause panic problem:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x7822e4]

This was referenced May 26, 2023
Closed
Closed
@ircrp
Copy link
Author

ircrp commented Jun 1, 2023

Hi @gzliudan, any news on this please ? I am free in the coming days to test any changes.

@gzliudan
Copy link
Collaborator

gzliudan commented Jun 1, 2023

I‘m testing a solution now. If it works fine, I will push codes to you tomorrow.

@gzliudan
Copy link
Collaborator

gzliudan commented Jun 5, 2023

I am still testing the solution now. It may take 1-2 weeks. You can try this temporary fix https://github.com/gzliudan/XDPoSChain/tree/fix-issue-268 if you're in a hurry.

@ircrp
Copy link
Author

ircrp commented Jun 5, 2023 •
edited
Loading

I am still testing the solution now. It may take 1-2 weeks. You can try this temporary fix https://github.com/gzliudan/XDPoSChain/tree/fix-issue-268 if you're in a hurry.

Hey @gzliudan thanks for coming back on this. Just checked out the branch and while the panic is no longer happening the issue is still present in terms of masternodes mismatch which is blocking the sync from progressing past block 516600. See the logs below

ERROR[06-05|14:36:50] Different masternodes                    number=516600
ERROR[06-05|14:36:50] Masternodes from checkpoint header       len=17
ERROR[06-05|14:36:50]                                          i=0 m=xdc065551F0dcAC6f00CAe11192D462db709bE3758c
ERROR[06-05|14:36:50]                                          i=1 m=xdc0eaa5c2F0929C95ED0280e8840A2416C4EF16E9d
ERROR[06-05|14:36:50]                                          i=2 m=xdc25c65b4b379ac37CF78357c4915f73677022eaFF
ERROR[06-05|14:36:50]                                          i=3 m=xdc4398241671b3Dd484FE3213A4fB7511f30e7D7C0
ERROR[06-05|14:36:50]                                          i=4 m=xdc595b8170eAF2e53e47cc20db47aD063a1e6e2C0f
ERROR[06-05|14:36:50]                                          i=5 m=xdc664c4a7B15d91B07c468162f535909114C038b91
ERROR[06-05|14:36:50]                                          i=6 m=xdc72fb467Ef6dA285b6B0D8f7A25AbD6049385d5fD
ERROR[06-05|14:36:50]                                          i=7 m=xdc7923B7f78dFfacBb3e1b886e2410630975384C7F
ERROR[06-05|14:36:50]                                          i=8 m=xdc7aA125338BE075260e77c6a66A56c90A5DeC4C58
ERROR[06-05|14:36:50]                                          i=9 m=xdc9a3787688fD210EC8F8D0224c6c50B8178d75Bc0
ERROR[06-05|14:36:50]                                          i=10 m=xdcA0caeC455E7585d845BBF72420397774e52f3B74
ERROR[06-05|14:36:50]                                          i=11 m=xdcA4F06BD6E4b217F1FfF9327BDE83da1d41193bE4
ERROR[06-05|14:36:50]                                          i=12 m=xdca65010026b83368Ca05DF6E8B467985D6de3eAC5
ERROR[06-05|14:36:50]                                          i=13 m=xdcC7d49d0a2CF198DeEBd6ce581aF465944ec8b2bB
ERROR[06-05|14:36:50]                                          i=14 m=xdccfCCDeA1006a5CfA7d9484B5b293B46964c265c0
ERROR[06-05|14:36:50]                                          i=15 m=xdcDb2E141595d8Edf6B1CE40b10d57Ca5b83855ec5
ERROR[06-05|14:36:50]                                          i=16 m=xdced2cE9d056DB8DeF2B4d877c9a80b6f988ae202E
ERROR[06-05|14:36:50] Signers from snapshot                    len=18
ERROR[06-05|14:36:50]                                          i=0  s=xdc0eaa5c2F0929C95ED0280e8840A2416C4EF16E9d
ERROR[06-05|14:36:50]                                          i=1  s=xdc25c65b4b379ac37CF78357c4915f73677022eaFF
ERROR[06-05|14:36:50]                                          i=2  s=xdc4398241671b3Dd484FE3213A4fB7511f30e7D7C0
ERROR[06-05|14:36:50]                                          i=3  s=xdc595b8170eAF2e53e47cc20db47aD063a1e6e2C0f
ERROR[06-05|14:36:50]                                          i=4  s=xdc664c4a7B15d91B07c468162f535909114C038b91
ERROR[06-05|14:36:50]                                          i=5  s=xdc72fb467Ef6dA285b6B0D8f7A25AbD6049385d5fD
ERROR[06-05|14:36:50]                                          i=6  s=xdc7923B7f78dFfacBb3e1b886e2410630975384C7F
ERROR[06-05|14:36:50]                                          i=7  s=xdc7aA125338BE075260e77c6a66A56c90A5DeC4C58
ERROR[06-05|14:36:50]                                          i=8  s=xdc9a3787688fD210EC8F8D0224c6c50B8178d75Bc0
ERROR[06-05|14:36:50]                                          i=9  s=xdcA0caeC455E7585d845BBF72420397774e52f3B74
ERROR[06-05|14:36:50]                                          i=10 s=xdcA4F06BD6E4b217F1FfF9327BDE83da1d41193bE4
ERROR[06-05|14:36:50]                                          i=11 s=xdca65010026b83368Ca05DF6E8B467985D6de3eAC5
ERROR[06-05|14:36:50]                                          i=12 s=xdcc67c2DEc79da735d6587d8DB3c23271d557196ab
ERROR[06-05|14:36:50]                                          i=13 s=xdcC7d49d0a2CF198DeEBd6ce581aF465944ec8b2bB
ERROR[06-05|14:36:50]                                          i=14 s=xdccfCCDeA1006a5CfA7d9484B5b293B46964c265c0
ERROR[06-05|14:36:50]                                          i=15 s=xdcDb2E141595d8Edf6B1CE40b10d57Ca5b83855ec5
ERROR[06-05|14:36:50]                                          i=16 s=xdced2cE9d056DB8DeF2B4d877c9a80b6f988ae202E
ERROR[06-05|14:36:50]                                          i=17 s=xdcFaa7A3c5b5a7f8c9A69f73347F9F28E2B9669d72
ERROR[06-05|14:36:50]
########## BAD BLOCK #########
Chain config: {ChainID: 50 Homestead: 1 DAO: <nil> DAOSupport: false EIP150: 2 EIP155: 3 EIP158: 3 Byzantium: 4 Constantinople: <nil> Engine: XDPoS}

Number: 516600
Hash: 0x0172e824a356fc03c7e6cd9f4f1bdac5d04f1f4a6dc4a652c36149b5a5902e1e


Error: Can't get signers from Smart Contract . Err: missing trie Node 19eb36f323caa1f7569ebed6fd78bb221c6db43f94d183526b4aca7f8958026a (path )
##############################

WARN [06-05|14:36:50] Synchronisation failed, dropping peer    peer=42a106de1aac361b err="retrieved hash chain is invalid"

Is that not the case on your end ?

@gzliudan
Copy link
Collaborator

gzliudan commented Jun 5, 2023 •
edited
Loading

Please remove your data dir /home/me/.xinfin first. Then add two parameters: --gcmode archive and --syncmode full in your command, such as:

./build/bin/XDC --datadir /home/me/.xinfin --gcmode archive --syncmode full --networkid 50 --etherbase 0x000000000000000000000000000000000000dead

The default value of parameter --gcmode is full, this causes your error when sync. We must set it to archive.

@ircrp
Copy link
Author

ircrp commented Jun 5, 2023

Please remove your data dir /home/me/.xinfin first. Then add two parameters: --gcmode archive and --syncmode full in your command, such as:

./build/bin/XDC --datadir /home/me/.xinfin --gcmode archive --syncmode full --networkid 50 --etherbase 0x000000000000000000000000000000000000dead

The default value of parameter --gcmode is full, this causes your error when sync. We must set it to archive.

Awesome thanks for the above instructions, I can now confirm I am now synced past block 516600 (more precisely currently at 1076850). Thank you !

@gzliudan
Copy link
Collaborator

gzliudan commented Jun 10, 2023 •
edited
Loading

I updated my fix: https://github.com/gzliudan/XDPoSChain/tree/fix-issue-268. Please add flag skip-signers in command, such as:

./build/bin/XDC \
    --datadir /home/me/.xinfin \
    --networkid 50 \
    --etherbase 0x000000000000000000000000000000000000dead \
    --gcmode archive \
    --syncmode full \
    --skip-signers

I'm testing this new fix now. I'll submit PR if no problem.

@gzliudan gzliudan mentioned this issue Jun 17, 2023
Closed
16 tasks
This was referenced Jul 6, 2023
Merged
Closed
@liam-lai liam-lai mentioned this issue Aug 28, 2023
Merged
@gzliudan
Copy link
Collaborator

Based on my recent study and tests, this issue occurs when we use golang v1.19 and above version. There is no such problem when I use golang under v1.19, such as v1.18.10. But the golang v1.14.15 is suggested since it is the version of most nodes. You can use the master or dev-upgrade code branch.

@wjrjerome
Copy link
Collaborator

hi @ircrp could you plz confirm if the above approach mentioned by @gzliudan has fixed your issue?

@gzliudan
Copy link
Collaborator

This problem has been solved since xdc chain support golang v1.21 now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@gzliudan @wjrjerome @ircrp