block/chunk corrupted: missing chunk part when restart lotus-miner #4045

decoqt · 2020-09-26T13:28:36Z

Describe the bug
ERROR: starting node: could not build arguments for function "github.com/filecoin-project/lotus/node/modules/lp2p".StartListening.func1 (/home/mcloud/ceph/current/lotus-0.8.0/node/modules/lp2p/addrs.go:98): failed to build host.Host: could not build arguments for function "reflect".makeFuncStub (/usr/local/go/src/reflect/asm_amd64.s:12): failed to build lp2p.BaseIpfsRouting: could not build arguments for function "reflect".makeFuncStub (/usr/local/go/src/reflect/asm_amd64.s:12): failed to build dtypes.MetadataDS: received non-nil error from function "reflect".makeFuncStub (/usr/local/go/src/reflect/asm_amd64.s:12): opening datastore /metadata: leveldb/journal: block/chunk corrupted: missing chunk part (0 bytes) [file=15008041.log]

To Reproduce
Steps to reproduce the behavior:

lotus-miner stop tooks too much time, then kill the process;
lotus-miner run got this log

Expected behavior
lotus-miner can run

Screenshots
If applicable, add screenshots to help explain your problem.

Version (run lotus version):
0.8.0

Additional context
Add any other context about the problem here.

The text was updated successfully, but these errors were encountered:

jennijuju · 2020-09-26T13:40:52Z

Was your disk space full?

decoqt · 2020-09-26T13:42:59Z

Was your disk space full?

no, 58% used

decoqt · 2020-09-26T17:06:31Z

Used leveldb to repair metadata, however, sst file in staging is missed also, therefore no sector list. So, I think we need a tool to reconstruct the metainfo of sectors from chain state. Otherwise, all sectors are lost.

jennijuju · 2020-09-30T17:56:47Z

Another one

s0nik42@keats:~$ lotus daemon start
2020-09-30T17:30:57.969+0200	INFO	main	lotus/daemon.go:170	lotus repo: /home/s0nik42/.lotus
2020-09-30T17:30:57.970+0200	INFO	build	[email protected]/paramfetch.go:138	Parameter file /var/tmp/filecoin-proof-parameters/v28-stacked-proof-of-replication-merkletree-poseidon_hasher-8-0-0-sha256_hasher-6babf46ce344ae495d558e7770a585b2382d54f225af8ed0397b8be7c3fcd472.vk is ok
2020-09-30T17:30:57.970+0200	INFO	build	[email protected]/paramfetch.go:138	Parameter file /var/tmp/filecoin-proof-parameters/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-0-0-5294475db5237a2e83c3e52fd6c2b03859a1831d45ed08c4f35dbf9a803165a9.vk is ok
2020-09-30T17:30:57.970+0200	INFO	build	[email protected]/paramfetch.go:138	Parameter file /var/tmp/filecoin-proof-parameters/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-8-0-559e581f022bb4e4ec6e719e563bf0e026ad6de42e56c18714a2c692b1b88d7e.vk is ok
2020-09-30T17:30:57.970+0200	INFO	build	[email protected]/paramfetch.go:138	Parameter file /var/tmp/filecoin-proof-parameters/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-0-0-50c7368dea9593ed0989e70974d28024efa9d156d585b7eea1be22b2e753f331.vk is ok
2020-09-30T17:30:57.970+0200	INFO	build	[email protected]/paramfetch.go:138	Parameter file /var/tmp/filecoin-proof-parameters/v28-stacked-proof-of-replication-merkletree-poseidon_hasher-8-8-0-sha256_hasher-82a357d2f2ca81dc61bb45f4a762807aedee1b0a53fd6c4e77b46a01bfef7820.vk is ok
2020-09-30T17:30:57.970+0200	INFO	build	[email protected]/paramfetch.go:138	Parameter file /var/tmp/filecoin-proof-parameters/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-0-0-3ea05428c9d11689f23529cde32fd30aabd50f7d2c93657c1d3650bca3e8ea9e.vk is ok
2020-09-30T17:30:57.970+0200	INFO	build	[email protected]/paramfetch.go:138	Parameter file /var/tmp/filecoin-proof-parameters/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-0-0-0cfb4f178bbb71cf2ecfcd42accce558b27199ab4fb59cb78f2483fe21ef36d9.vk is ok
2020-09-30T17:30:57.970+0200	INFO	build	[email protected]/paramfetch.go:138	Parameter file /var/tmp/filecoin-proof-parameters/v28-stacked-proof-of-replication-merkletree-poseidon_hasher-8-0-0-sha256_hasher-ecd683648512ab1765faa2a5f14bab48f676e633467f0aa8aad4b55dcb0652bb.vk is ok
2020-09-30T17:30:57.971+0200	INFO	build	[email protected]/paramfetch.go:138	Parameter file /var/tmp/filecoin-proof-parameters/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-8-2-b62098629d07946e9028127e70295ed996fe3ed25b0f9f88eb610a0ab4385a3c.vk is ok
2020-09-30T17:30:57.970+0200	INFO	build	[email protected]/paramfetch.go:138	Parameter file /var/tmp/filecoin-proof-parameters/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-0-0-0170db1f394b35d995252228ee359194b13199d259380541dc529fb0099096b0.vk is ok
2020-09-30T17:30:57.970+0200	INFO	build	[email protected]/paramfetch.go:138	Parameter file /var/tmp/filecoin-proof-parameters/v28-stacked-proof-of-replication-merkletree-poseidon_hasher-8-8-2-sha256_hasher-96f1b4a04c5c51e4759bbf224bbc2ef5a42c7100f16ec0637123f16a845ddfb2.vk is ok
2020-09-30T17:30:57.971+0200	INFO	build	[email protected]/paramfetch.go:138	Parameter file /var/tmp/filecoin-proof-parameters/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-0-0-7d739b8cf60f1b0709eeebee7730e297683552e4b69cab6984ec0285663c5781.vk is ok
2020-09-30T17:30:57.971+0200	INFO	build	[email protected]/paramfetch.go:138	Parameter file /var/tmp/filecoin-proof-parameters/v28-stacked-proof-of-replication-merkletree-poseidon_hasher-8-0-0-sha256_hasher-032d3138d22506ec0082ed72b2dcba18df18477904e35bafee82b3793b06832f.vk is ok
2020-09-30T17:30:57.977+0200	INFO	build	[email protected]/paramfetch.go:138	Parameter file /var/tmp/filecoin-proof-parameters/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-8-0-0377ded656c6f524f1618760bffe4e0a1c51d5a70c4509eedae8a27555733edc.vk is ok
2020-09-30T17:30:57.978+0200	INFO	build	[email protected]/paramfetch.go:138	Parameter file /var/tmp/filecoin-proof-parameters/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-8-2-2627e4006b67f99cef990c0a47d5426cb7ab0a0ad58fc1061547bf2d28b09def.vk is ok
2020-09-30T17:30:57.978+0200	INFO	build	[email protected]/paramfetch.go:162	parameter and key-fetching complete
2020-09-30T17:30:57.998+0200	INFO	badger	[email protected]/levels.go:183	All 0 tables opened in 0s
2020-09-30T17:30:58.008+0200	INFO	badger	[email protected]/value.go:1158	Replaying file id: 0 at offset: 0
2020-09-30T17:30:58.008+0200	INFO	badger	[email protected]/value.go:1178	Replay took: 3.37µs
2020-09-30T17:30:58.010+0200	INFO	badger	[email protected]/levels.go:183	All 1 tables opened in 0s
2020-09-30T17:30:58.014+0200	INFO	badger	[email protected]/value.go:1158	Replaying file id: 0 at offset: 737367
2020-09-30T17:30:58.014+0200	INFO	badger	[email protected]/value.go:1178	Replay took: 64.33µs
2020-09-30T17:31:01.048+0200	INFO	badger	[email protected]/levels.go:129	890 tables out of 1186 opened in 3s
2020-09-30T17:31:02.175+0200	INFO	badger	[email protected]/levels.go:183	All 1186 tables opened in 4.127s
2020-09-30T17:31:02.189+0200	INFO	badger	[email protected]/value.go:1158	Replaying file id: 439 at offset: 702642107
2020-09-30T17:31:02.195+0200	INFO	badger	[email protected]/value.go:1178	Replay took: 6.364471ms
2020-09-30T17:31:02.211+0200	WARN	modules	modules/chain.go:95	loading chain state from disk: loading tipset: get block bafy2bzacecywz5uctlwsdtfdytekuzo74pzifdgtpnsft4b3kbqixeizcj5fo: blockstore: block not found
2020-09-30T17:31:02.237+0200	INFO	p2pnode	lp2p/addrs.go:114	Swarm listening at: [/ip4/127.0.0.1/tcp/42001 /ip4/192.168.43.1/tcp/42001 /ip4/172.17.0.1/tcp/42001]
2020-09-30T17:31:02.239+0200	INFO	drand	log/log.go:109		{"level": "info", "optimizing_client": "watching on client", "client": "HTTP(\"https://pl-eu.incentinet.drand.sh/\").(+verifier)"}
2020-09-30T17:31:02.240+0200	INFO	drand	log/log.go:109		{"level": "info", "optimizing_client": "watching on client", "client": "HTTP(\"https://api.drand.sh/\").(+verifier)"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x105ec9c]
goroutine 1 [running]:
github.com/filecoin-project/lotus/chain/types.(*TipSet).Cids(...)
	/home/s0nik42/lotus/chain/types/tipset.go:140
github.com/filecoin-project/lotus/chain/stmgr.(*StateManager).TipSetState(0xc0c5c06800, 0x2f91a80, 0xc0a88a6030, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/home/s0nik42/lotus/chain/stmgr/stmgr.go:77 +0x10c
github.com/filecoin-project/lotus/chain/messagepool.(*mpoolProvider).GetActorAfter(0xc0c5ebce00, 0xc0c5eba500, 0x31, 0x0, 0x61, 0x0, 0x0)
	/home/s0nik42/lotus/chain/messagepool/provider.go:50 +0x57
github.com/filecoin-project/lotus/chain/messagepool.(*MessagePool).getStateNonce(0xc000158580, 0xc0c5eba500, 0x31, 0x0, 0xc0c4ff67c0, 0x2f54640, 0xc0c6463e90)
	/home/s0nik42/lotus/chain/messagepool/messagepool.go:781 +0x56
github.com/filecoin-project/lotus/chain/messagepool.(*MessagePool).addLoaded(0xc000158580, 0xc0c4ff6750, 0x0, 0x0)
	/home/s0nik42/lotus/chain/messagepool/messagepool.go:660 +0xf0
github.com/filecoin-project/lotus/chain/messagepool.(*MessagePool).loadLocal(0xc000158580, 0xc0c5ebce40, 0x0)
	/home/s0nik42/lotus/chain/messagepool/messagepool.go:1302 +0x308
github.com/filecoin-project/lotus/chain/messagepool.New(0x2fac280, 0xc0c5ebce00, 0x7fbaa8de5358, 0xc0001bb200, 0x2b5d2b6, 0xa, 0xc0c6329920, 0x60, 0xc000648640)
	/home/s0nik42/lotus/chain/messagepool/messagepool.go:358 +0x7bd
github.com/filecoin-project/lotus/node/modules.MessagePool(0x2f57e80, 0xc000434b28, 0xc0c5c06800, 0xc015580300, 0x7fbaa8de5358, 0xc0001bb200, 0x2b5d2b6, 0xa, 0x0, 0x0, ...)
	/home/s0nik42/lotus/node/modules/chain.go:56 +0xa3
reflect.Value.call(0x28cc400, 0x2d0cbb0, 0x13, 0x2b53c6e, 0x4, 0xc00076e480, 0x5, 0x5, 0x203000, 0x5265b6, ...)
	/usr/local/go/src/reflect/value.go:460 +0x8ab
reflect.Value.Call(0x28cc400, 0x2d0cbb0, 0x13, 0xc00076e480, 0x5, 0x5, 0x8, 0x10, 0xc0c5ebcdf0)
	/usr/local/go/src/reflect/value.go:321 +0xb4
github.com/filecoin-project/lotus/node.as.func2(0xc00076e480, 0x5, 0x5, 0xc00076e480, 0xc000801640, 0x295b420)
	/home/s0nik42/lotus/node/options.go:140 +0xf0
reflect.callReflect(0xc000500990, 0xc000648b20, 0xc000648b08)
	/usr/local/go/src/reflect/value.go:549 +0x322
reflect.makeFuncStub(0x2f57e80, 0xc000434b28, 0xc0c5c06800, 0xc015580300, 0x7fbaa8de5358, 0xc0001bb200, 0x2b5d2b6, 0xa, 0x0, 0x0, ...)
	/usr/local/go/src/reflect/asm_amd64.s:20 +0x42
reflect.Value.call(0x28cc400, 0xc000500990, 0x13, 0x2b53c6e, 0x4, 0xc00076e400, 0x5, 0x5, 0x10, 0xc0c5ebcdc0, ...)
	/usr/local/go/src/reflect/value.go:460 +0x8ab
reflect.Value.Call(0x28cc400, 0xc000500990, 0x13, 0xc00076e400, 0x5, 0x5, 0x2953e00, 0x2a9de01, 0xc0c5ebcdc0)
	/usr/local/go/src/reflect/value.go:321 +0xb4
go.uber.org/dig.defaultInvoker(0x28cc400, 0xc000500990, 0x13, 0xc00076e400, 0x5, 0x5, 0xc0000da230, 0xc00076e400, 0x5)
	/home/s0nik42/go/pkg/mod/go.uber.org/[email protected]/dig.go:284 +0x64
go.uber.org/dig.(*node).Call(0xc00045f5e0, 0x2fac820, 0xc0000da230, 0x2fc4840, 0x2b2b160)
	/home/s0nik42/go/pkg/mod/go.uber.org/[email protected]/dig.go:710 +0x269
go.uber.org/dig.paramSingle.Build(0x0, 0x0, 0x0, 0x2fc4840, 0x2b2b160, 0x2fac820, 0xc0000da230, 0x2b1a000, 0xc0c4ff20a8, 0x96, ...)
	/home/s0nik42/go/pkg/mod/go.uber.org/[email protected]/param.go:245 +0x2e0
go.uber.org/dig.paramList.BuildList(0x2fc4840, 0x28b89c0, 0xc0c6329860, 0x6, 0x6, 0x2fac820, 0xc0000da230, 0x0, 0x0, 0x58cd42, ...)
	/home/s0nik42/go/pkg/mod/go.uber.org/[email protected]/param.go:201 +0xb8
go.uber.org/dig.(*Container).Invoke(0xc0000da230, 0x28b89c0, 0x2d0cb98, 0x0, 0x0, 0x0, 0x0, 0x0)
	/home/s0nik42/go/pkg/mod/go.uber.org/[email protected]/dig.go:432 +0x2ba
go.uber.org/fx.(*App).executeInvokes(0xc0004b62c0, 0x27f4800, 0xc000800250)
	/home/s0nik42/go/pkg/mod/go.uber.org/[email protected]/app.go:547 +0x368
go.uber.org/fx.New(0xc0006416a8, 0x3, 0x3, 0xc0003af040)
	/home/s0nik42/go/pkg/mod/go.uber.org/[email protected]/app.go:345 +0x4a8
github.com/filecoin-project/lotus/node.New(0x2f91a80, 0xc0005bccc0, 0xc00011a000, 0x8, 0x8, 0x0, 0x0, 0x0)
	/home/s0nik42/lotus/node/builder.go:546 +0x47c
main.glob..func2(0xc0005c27c0, 0x0, 0x0)
	/home/s0nik42/lotus/cmd/lotus/daemon.go:238 +0xe64
github.com/urfave/cli/v2.(*App).RunAsSubcommand(0xc000714480, 0xc0005c2380, 0x0, 0x0)
	/home/s0nik42/go/pkg/mod/github.com/urfave/cli/[email protected]/app.go:432 +0x993
github.com/urfave/cli/v2.(*Command).startApp(0x5305a40, 0xc0005c2380, 0x7ffe39cb85fe, 0x6)
	/home/s0nik42/go/pkg/mod/github.com/urfave/cli/[email protected]/command.go:279 +0x68c
github.com/urfave/cli/v2.(*Command).Run(0x5305a40, 0xc0005c2380, 0x0, 0x0)
	/home/s0nik42/go/pkg/mod/github.com/urfave/cli/[email protected]/command.go:94 +0xa12
github.com/urfave/cli/v2.(*App).RunContext(0xc000714300, 0x2f91a00, 0xc000040090, 0xc00003c180, 0x3, 0x3, 0x0, 0x0)
	/home/s0nik42/go/pkg/mod/github.com/urfave/cli/[email protected]/app.go:306 +0x814
github.com/urfave/cli/v2.(*App).Run(...)
	/home/s0nik42/go/pkg/mod/github.com/urfave/cli/[email protected]/app.go:215
github.com/filecoin-project/lotus/cli.RunApp(0xc000714300)
	/home/s0nik42/lotus/cli/helper.go:34 +0x7c
main.main()
	/home/s0nik42/lotus/cmd/lotus/main.go:73 +0x631

ERROR: starting node: could not build arguments for function “github.com/filecoin-project/lotus/node/modules/lp2p”.StartListening.func1 (/home/s0nik42/lotus/node/modules/lp2p/addrs.go:98): failed to build host.Host: could not build arguments for function “reflect”.makeFuncStub (/usr/local/go/src/reflect/asm_amd64.s:12): failed to build lp2p.BaseIpfsRouting: could not build arguments for function “reflect”.makeFuncStub (/usr/local/go/src/reflect/asm_amd64.s:12): failed to build dtypes.MetadataDS: received non-nil error from function “reflect”.makeFuncStub (/usr/local/go/src/reflect/asm_amd64.s:12): opening datastore /metadata: leveldb/journal: block/chunk corrupted: chunk length overflows block (366 bytes) [file=000253.log]

apengn · 2021-03-20T01:19:31Z

3月 20 01:16:41 192-168-160-190 lotus-miner[288303]: ERROR: creating node: starting node: could not build arguments for function "github.com/filecoin-project/lotus/node/modules/lp2p".StartListening.func1 (/home/fil/wp/lotus/node/modules/lp2p/addrs.go:98): failed to build host.Host: could not build arguments for function "reflect".makeFuncStub (/usr/local/go/src/reflect/asm_amd64.s:12): failed to build lp2p.RawHost: could not build arguments for function "reflect".makeFuncStub (/usr/local/go/src/reflect/asm_amd64.s:12): could not build value group []config.Option[group="libp2p"]: could not build arguments for function "github.com/filecoin-project/lotus/node/modules/lp2p".ConnGaterOption (/home/fil/lotus/node/modules/lp2p/conngater.go:14): failed to build *conngater.BasicConnectionGater: could not build arguments for function "reflect".makeFuncStub (/usr/local/go/src/reflect/asm_amd64.s:12): failed to build dtypes.MetadataDS: received non-nil error from function "reflect".makeFuncStub (/usr/local/go/src/reflect/asm_amd64.s:12): opening datastore /metadata: leveldb/journal: block/chunk corrupted: chunk length overflows block (4016 bytes) [file=005107.log] 3月 20 01:16:41 192-168-160-190 systemd[1]: lotus-miner-sync.service: Main process exited, code=exited, status=1/FAILURE

potato1992 · 2021-09-21T23:59:19Z

FYI, I faced the same issue. And I just solved it by editing the code to stop loading at the point of corruption so that I can recover the data as much as I have.

TippyFlitsUK · 2022-04-29T11:58:47Z

Hi @yydfjt

Thanks for the report. The issue is outdated.

If this is an issue in the current release - please open a new ticket.

Thank you !

This was referenced Sep 30, 2020

Rebuild miner database(metadata) after a corruption #4075

Closed

Lotus datastore corruption when running out of space #4123

Closed

TippyFlitsUK closed this as completed Apr 29, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

block/chunk corrupted: missing chunk part when restart lotus-miner #4045

block/chunk corrupted: missing chunk part when restart lotus-miner #4045

decoqt commented Sep 26, 2020

jennijuju commented Sep 26, 2020 •

edited

Loading

decoqt commented Sep 26, 2020

decoqt commented Sep 26, 2020

jennijuju commented Sep 30, 2020

apengn commented Mar 20, 2021 •

edited

Loading

potato1992 commented Sep 21, 2021

TippyFlitsUK commented Apr 29, 2022

block/chunk corrupted: missing chunk part when restart lotus-miner #4045

block/chunk corrupted: missing chunk part when restart lotus-miner #4045

Comments

decoqt commented Sep 26, 2020

jennijuju commented Sep 26, 2020 • edited Loading

decoqt commented Sep 26, 2020

decoqt commented Sep 26, 2020

jennijuju commented Sep 30, 2020

apengn commented Mar 20, 2021 • edited Loading

potato1992 commented Sep 21, 2021

TippyFlitsUK commented Apr 29, 2022

jennijuju commented Sep 26, 2020 •

edited

Loading

apengn commented Mar 20, 2021 •

edited

Loading