[BUG] Waste space in lotus Home.

[s0nik42]

Describe the bug

I'm creating this issue to Highlight that small individual bugs that are consuming space in the lotus home folder start to be a big problem when they all together consume more than 100GiB/day additionnal space
We almost have to clean up the .lotus and .lotusminer every 2 days to avoid being out of space.

Version (run lotus version):
Daemon: 1.8.0+mainnet+git.5c56017db+api1.2.0
Local: lotus version 1.8.0+mainnet+git.5c56017db

1/ .lotus/journal (15GiB/day) #5291
2/ .lotus/kvlog/metadata/*.cbor (8GiB/day)
3/ .lotusminer/kvlog/metadata/*.cbor (8GiB/day)
4/ .lotusminer/fstmp (32GiB for each car file imported via offline deals) #5291 #5591
5/ .lotus/datastore/client (2GiB 34GiB)
36/ .lotus/datastore/staging (seems to be solved) #5569
6/ .lotusminer/datastore/staging (489G)
7/ lotus-miner backup : > 1.3GiB

Expected behavior
Should clean/produce less data automatically without having to restart and manually clean them

PS : @jennijuju @ribasushi . We discussed that 2 weeks ago during our minerX call.

[stuberman]

I also see most of these issues (now running 1.9.0-rc1)
I want to add another two clean up items:
7. After sector upgrade (CC below), original sector is no longer active but still have 32GiB remaining
8. After all deals in a sector expire, the sectors remain in NOT ACTIVE state but continue to be stored (see 137-156)

lotus-miner sectors list |grep NO

0 Proving YES NO 1554348 (in 41 weeks 5 days) CC
1 Proving YES NO 1554348 (in 41 weeks 5 days) CC
2 Proving YES NO 1554348 (in 41 weeks 5 days) CC
45 Proving YES NO 1594668 (in 43 weeks 5 days) CC
137 Proving YES NO 647148 (3 weeks 1 day ago) 7 5.035GiB
143 Proving YES NO 652908 (2 weeks 6 days ago) 8 13.42GiB
144 Proving YES NO 652908 (2 weeks 6 days ago) 8 1.97GiB
145 Proving YES NO 655788 (2 weeks 5 days ago) 6 2.457GiB
146 Proving YES NO 655788 (2 weeks 5 days ago) 1 251.8MiB
151 Proving YES NO 661548 (2 weeks 3 days ago) 12 4.426GiB
153 Proving YES NO 664428 (2 weeks 2 days ago) 9 15.78GiB
155 Proving YES NO 667308 (2 weeks 1 day ago) 3 1.721GiB
156 Proving YES NO 667308 (2 weeks 1 day ago) 3 4.428GiB

ls -Alh /pool/storage/sealed |grep s-t01278-156
-rw-r--r-- 1 stuart stuart 32G Jan 7 10:36 s-t01278-156
ls -Alh /pool/storage/unsealed |grep s-t01278-156
-rw-r--r-- 1 stuart stuart 33G Oct 11 2020 s-t01278-156
ls -Alh /pool/storage/cache |grep s-t01278-156
drwxr-xr-x 2 stuart stuart 12 Jan 7 16:34 s-t01278-156
ls -Alh /pool/storage/cache/s-t01278-156
total 74M
-rw-rw-r-- 1 stuart stuart 64 Jan 7 16:34 p_aux
-rw-rw-r-- 1 stuart stuart 9.2M Jan 7 16:34 sc-02-data-tree-r-last-0.dat
-rw-rw-r-- 1 stuart stuart 9.2M Jan 7 16:34 sc-02-data-tree-r-last-1.dat
-rw-rw-r-- 1 stuart stuart 9.2M Jan 7 16:34 sc-02-data-tree-r-last-2.dat
-rw-rw-r-- 1 stuart stuart 9.2M Jan 7 16:34 sc-02-data-tree-r-last-3.dat
-rw-rw-r-- 1 stuart stuart 9.2M Jan 7 16:34 sc-02-data-tree-r-last-4.dat
-rw-rw-r-- 1 stuart stuart 9.2M Jan 7 16:34 sc-02-data-tree-r-last-5.dat
-rw-rw-r-- 1 stuart stuart 9.2M Jan 7 16:34 sc-02-data-tree-r-last-6.dat
-rw-rw-r-- 1 stuart stuart 9.2M Jan 7 16:34 sc-02-data-tree-r-last-7.dat
-rw-rw-r-- 1 stuart stuart 1020 Jan 7 16:34 t_aux

ls -Alh /pool/storage/sealed |grep s-t01278-45
-rw-r--r-- 1 stuart stuart 32G Jan 7 08:33 s-t01278-45

[jennijuju]

Why do you think this is a bug? or this is more like an improvements request?

[jennijuju]

Implement automatic compaction without restart

[s0nik42]

@jennijuju not sure if you are asking this to @stuberman or me.
ghost fstmp files are definitely a bug , same for the journal folder. But I'm happy with any tagging as long as its solved :p

[s0nik42]

Original Ticket updated to reflect current status

[s0nik42]

my kvlog is 100GiB under v1.11.1-rc1 , insanely big. I upgrade to v1.11.2-rc2 and let you know.

[s0nik42]

my kvlog is 900GiB under 1.13.0+mainnet+git.7a55e8e89

[RobQuistNL]

just went to 1.1TB and broke my miner node. ( 1.13.2-rc4 )
Again

[stuberman]

Here are the cbor sizes on my machine - I have not touched them in many months

ls -Alh /pool/storage/kvlog/metadata/*.cbor

-rw-r--r-- 1 stuart stuart 5.5G Jan 22 13:26 /pool/storage/kvlog/metadata/1622657491.log.cbor (lotus-miner)

ls -Alh /.lotus/kvlog/metadata/*.cbor

-rw-r--r-- 1 stuart stuart 51M Jan 22 13:32 /.lotus/kvlog/metadata/1642432663.log.cbor

lotus-miner version

Daemon: 1.13.2-rc4+mainnet+git.e05fdf892+api1.2.0
Local: lotus-miner version 1.13.2-rc4+mainnet+git.e05fdf892

lotus-miner/config.toml

[Backup]
DisableMetadataLog = false

[RobQuistNL]

Its usually normal. But sometimes something happens and lotus goes haywire and stuffs that log with data, filling up my disk, breaking all other log files because of 0 diskspace.

[rjan90]

Hey @s0nik42!

Thank you so much for this comprehensive list of issues regarding waste space.. But I think it´s time to close this and rather create seperate tickets for the rest of these things, so it´s easier to keep track of and easier to surface.

NB: The most pressing of these issues as I see it: the looping logs causing the kvlog to become large really fast has been fixed in #8338

Thanks again!