zstdchunked error trying to delete uncompressed big layers

[evkogs]

Hi. Error happens both on latest release and main branch:

INFO:Waitingforapplicationshutdown. 2024-10-2822:05:44,029-artisan_generate_v2-INFO-[artisan_generate_v2.py:965]-Startinggracefulshutdown... 2024-10-2822:05:44,030-artisan_generate_v2-INFO-[artisan_generate_v2.py:446]-Generate:Tasksstopped 2024-10-2822:05:44,030-artisan_generate_v2-INFO-[artisan_generate_v2.py:446]-Analyze:Tasksstopped 2024-10-2822:05:44,030-artisan_generate_v2-INFO-[artisan_generate_v2.py:993]-Serviceshutdowncompletedsuccessfully INFO:Applicationshutdowncomplete. INFO:Finishedserverprocess[1] INFO[0280]containerexitwithcode0 INFO[0281]converting...digest="sha256:a27d598af88b3e82c1bc0419751e255bc7e156473aa7617d48ec4a17552a9e3d" INFO[0281]converting...digest="sha256:5c9d6bf22309e10584674f86f8c418a13164dced70ed937c957e684f0582aa3a" INFO[0281]converting...digest="sha256:2c1ce468d9f3d941396801f6e3afc8921466650dd05430fe644cd3537713d27f" INFO[0281]converting...digest="sha256:a18c6c94bee1c54df04ccb9eb97ab14e654c92a16f80bb957369e355d20d7b9b" INFO[0281]converting...digest="sha256:25b9a558206373e8d08f56bc08800c483ac459cd0baf6fb355c5eb10af3f9492" INFO[0281]converting...digest="sha256:56ad0ef4a23a1ace287a75ec76ae3a51b8558b612286e888b29c099bfa2bbdf5" INFO[0281]converting...digest="sha256:cffbcfe714048b18a6140e357230244af37bf295b42a3c2db78f0f1ff619f53b" INFO[0281]converting...digest="sha256:06ce3025ef841e9964942b982bbd1bf18da04710619058282b07a50fa1f288fb" INFO[0281]converting...digest="sha256:4769f6a754c921905b320b9bae0e7aa061310404d16bfacebf5858cf38fed2e6" INFO[0281]converting...digest="sha256:5172feb1975ff9ceb946a70cdb56e40fdccc31266ba74e7193318242f93911ac" INFO[0281]converting...digest="sha256:da6fcb7a1c33ae01cbc5ced784e79222e4c8c1eefba4987d9d0f93f36a97c95a" INFO[0281]converting...digest="sha256:2fa3e9c98d9ebf6ddb46d548dd22f018b35abd959232e08c01412488d4daf13c" INFO[0281]converting...digest="sha256:1ee80b3491efc3953fc908d21e855e1a2500bccb3ba9cb0fcf571219e43240a3" INFO[0281]converting...digest="sha256:9f1bc10187e18e4e3f23878b0e675d9bce3fbb6f63a92e7b9cd7e27d1072c0da" INFO[0281]converting...digest="sha256:fecc4f17ced3eef45a9549aa42e90057b5bb23793aebcf706e66e7f244ec12ae" INFO[0281]converting...digest="sha256:c4592752d26e17a8052fcbc81feb356b4d635d7d2e658270f2e702c6aca9b08d" INFO[0281]converting...digest="sha256:1b65aada3dc7530818fe66ace7861323fb14dad2e887a35c8fae526268185b8d" INFO[0281]converting...digest="sha256:d5337dc44a98a5d699fcbf5cb4a902d7ce4f2f9efccc4575538c05023d859c22" INFO[0281]converting...digest="sha256:0dc726ffa0c80206fb049a0d55652be59d0879a030b15b956ed05750c3019533" INFO[0281]converting...digest="sha256:fdaa897852cb684c160d37e60f00a87926c21ea6eff99e3e8759184232f82d84" INFO[0281]converting...digest="sha256:a7f3c0c5e46c0359d31ac1e15b49344c7b5c00ad54ae9acd78352716e2cfe6ab" INFO[0281]converting...digest="sha256:6eff0b4725a00315bc9d7a10c1b4915695bf94ba909abb7869beb15d7b151e66" INFO[0281]converting...digest="sha256:2c1ce468d9f3d941396801f6e3afc8921466650dd05430fe644cd3537713d27f" INFO[0281]converting...digest="sha256:bfe4699a54e8ff1e8ef6491680fdb2de7c079142785915cd0eae7beaa3b42212" INFO[0281]converting...digest="sha256:2c1ce468d9f3d941396801f6e3afc8921466650dd05430fe644cd3537713d27f" INFO[0281]converting...digest="sha256:754f05bf73f68ccce52cca98f8640e3359a0d20b7f01c2a8b91e9737590ce577" INFO[0281]converting...digest="sha256:3212db16dc65f0724f7891287708a295c3b7d96fb0b9dd1def9d8a38dbb08e0a" INFO[0281]converting...digest="sha256:61a82ed9fa8485bf81bf4bc642425f85265860e7b65cd2ce9f7fc68ddb695991" INFO[0281]converting...digest="sha256:18756305808c3339ab9d63eda09ebdbe752ddd18a7cf44a38ceffd43d7a39c88" INFO[0281]converting...digest="sha256:62126182f43df62583967657a26faa869580411a6bb45760c322eb9904615e8b" INFO[0281]converting...digest="sha256:df10b8dd82572105005ae51ec4376536667bc63ccc6dfa1befa144f87d377cdf" INFO[0281]converting...digest="sha256:4b6d700862f3c973372dda536648a6628dc229f1293012edadd1137cacb7d633" INFO[0281]converting...digest="sha256:4aeba21c5b0e0ca25aab42897fc1180230a1a1c38e768773d6a17e9947967e12" INFO[0281]converting...digest="sha256:5fb5afe511614179132d099cc5a20779cca79743c8f0ef6a42b5755e301c4ae7" INFO[0281]converting...digest="sha256:2c1ce468d9f3d941396801f6e3afc8921466650dd05430fe644cd3537713d27f" INFO[0281]converting...digest="sha256:1647b7bb108c98aff18a5d949ee03abe638663635a5f291732b0ee0e50f26a8a" INFO[0281]converting...digest="sha256:2c1ce468d9f3d941396801f6e3afc8921466650dd05430fe644cd3537713d27f" INFO[0281]converting...digest="sha256:9410e61c5619dd663ae7ec545cbec2a3d8b761f91acbb70620600b3761d144d9" INFO[0281]converting...digest="sha256:c34d3e5b330ab0c975605b925b101ab10bef571b3e3be78f808318c4141b82d6" INFO[0281]converting...digest="sha256:41788e8c0c544f00192be6a2f208e550c4fe358890264d28ef0c29dd9650b413" INFO[0281]converting...digest="sha256:4b44da1896473e444070380ec5e2081a0605edca35a6857fbea118fb02c67f09" INFO[0281]converting...digest="sha256:ee55e14efd13610e1f098069f560f1587fc28bbe0d1fbc6239c113c825b9f962" INFO[0281]converting...digest="sha256:079852333815657265c55c55437c38c1bfb850aab755a1c6515a57fd8cb8a080" INFO[0281]converting...digest="sha256:bcf4bbb4a7eac7d68090c15e2db3b7d579a004561991e2b09c92f42f7066886d" INFO[0281]converting...digest="sha256:4659278455e77414a00f81d1036566113ba5da209e1acbd25cb7240734dcaa8d" INFO[0281]converting...digest="sha256:6fdca3926ba8ac490c4d510f4fcee464f664fb06a3685ab56c83a0ed4f48cc65" INFO[0281]converting...digest="sha256:cab72bee6c27ffb8aedad9c1dda5cd843be872ff4233bc89d3d115e42ac45685" INFO[0281]converting...digest="sha256:4b8e0d9ae834c23501db1b0cd99c6122a30d95f72d48efba446bfb99fe3348bb" INFO[0281]converting...digest="sha256:8a0eb9f389f6402922cd206197a86d176d86deb2a696e4fab0ef9cfa95c9607d" INFO[0281]converting...digest="sha256:135511128aeda993ff5199ae5eca498a914f32a0d7c126ce2c4210cf986d8bcd" INFO[0281]converting...digest="sha256:d449b3d0d8a15680505173497a56f067b4d5c0766f75ddde0eff2bf4de1514fb" INFO[0281]converting...digest="sha256:81a6de6d490b6fad5932ff4ed7a2c585b4fa03654ef893046af950a21099b6cb" INFO[0281]converting...digest="sha256:2c1ce468d9f3d941396801f6e3afc8921466650dd05430fe644cd3537713d27f" WARN[0287]failedtoremovetmpuncompressedlayererror="contextcanceled"uncompressedDesc="&{application/vnd.oci.image.layer.v1.tarsha256:2e69fe729d5788239a3713310c27ed5af34147e2b4a1df6f25ddb9dd440ba66a11264[]map[][]<nil>}" WARN[0287]failedtoremovetmpuncompressedlayererror="contextcanceled"uncompressedDesc="&{application/vnd.oci.image.layer.v1.tarsha256:9073450b516f979f3ae63598ee8e12fd8ca6460e02fda53da70b2871641b2b4d42496[]map[][]<nil>}" WARN[0287]failedtoremovetmpuncompressedlayererror="contextcanceled"uncompressedDesc="&{application/vnd.oci.image.layer.v1.tarsha256:69002f0165290f934255c49e3b5d58a26088445340f2843ee1959723b6ea6fea340992[]map[][]<nil>}" WARN[0287]failedtoremovetmpuncompressedlayererror="contextcanceled"uncompressedDesc="&{application/vnd.oci.image.layer.v1.tarsha256:78d295a0c8e004e84b10c4dfafe83385e57de34965443211c2ad0faabf216c4799840[]map[][]<nil>}" WARN[0287]failedtoremovetmpuncompressedlayererror="contextcanceled"uncompressedDesc="&{application/vnd.oci.image.layer.v1.tarsha256:43318f0006477ee6a02be460ab98c75e45bc23e2643b9f877b041b88a7cea17526112[]map[][]<nil>}" WARN[0287]failedtoremovetmpuncompressedlayererror="contextcanceled"uncompressedDesc="&{application/vnd.oci.image.layer.v1.tarsha256:a62c8c75f8c8b0841061d8b0d3589f77756cef54f4bb6e529a83cc1b0412ee2517408[]map[][]<nil>}" WARN[0287]failedtoremovetmpuncompressedlayererror="contextcanceled"uncompressedDesc="&{application/vnd.oci.image.layer.v1.tarsha256:14efc3c96ab2c5c0920c20a484d9755ea28e0b230310e7f64b7c7a0d31d998423072[]map[][]<nil>}" WARN[0287]failedtoremovetmpuncompressedlayererror="contextcanceled"uncompressedDesc="&{application/vnd.oci.image.layer.v1.tarsha256:1656497b35dd71fe1bd45aa1cdf38a6e49d035ab8efdf30b2124563a2f3faec33072[]map[][]<nil>}" WARN[0287]failedtoremovetmpuncompressedlayererror="contextcanceled"uncompressedDesc="&{application/vnd.oci.image.layer.v1.tarsha256:5115d7b09e2af6f16a520081cdbb07a01970bed93b71992d0da5018b53555ca58704[]map[][]<nil>}" WARN[0287]failedtoremovetmpuncompressedlayererror="contextcanceled"uncompressedDesc="&{application/vnd.oci.image.layer.v1.tarsha256:51c96fce3f2585d9ff3bab85bc3d8186b27610cd9807b4f490ebc4184aeb805530208[]map[][]<nil>}" WARN[0287]failedtoremovetmpuncompressedlayererror="contextcanceled"uncompressedDesc="&{application/vnd.oci.image.layer.v1.tarsha256:0f5d1fc9b9ac6c0541d19729fd70c74bf96fb0e21cb1b512179627ca43ecc6603584[]map[][]<nil>}" WARN[0287]failedtoremovetmpuncompressedlayererror="contextcanceled"uncompressedDesc="&{application/vnd.oci.image.layer.v1.tarsha256:f97588faa6c468724eda134f66eff4dca51a5f55bf5a4606fba8193ecdd474ca3584[]map[][]<nil>}" WARN[0287]failedtoremovetmpuncompressedlayererror="contextcanceled"uncompressedDesc="&{application/vnd.oci.image.layer.v1.tarsha256:0e256f201c5067cc84d79e2f9cc03d30937728fe791d43ecfa831cc00c7b7fb42048[]map[][]<nil>}" WARN[0287]failedtoremovetmpuncompressedlayererror="contextcanceled"uncompressedDesc="&{application/vnd.oci.image.layer.v1.tarsha256:edeb577b0f8827769d6a5a69adae1a235354afff23e4948ce0c6c2df2fa04743151040[]map[][]<nil>}" WARN[0287]failedtoremovetmpuncompressedlayererror="contextcanceled"uncompressedDesc="&{application/vnd.oci.image.layer.v1.tarsha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef1024[]map[][]<nil>}" WARN[0287]failedtoremovetmpuncompressedlayererror="contextcanceled"uncompressedDesc="&{application/vnd.oci.image.layer.v1.tarsha256:99f06a613f96be986a20d1dfa8e78d228bda97e58e5c09b53321407240d00c928704[]map[][]<nil>}" WARN[0287]failedtoremovetmpuncompressedlayererror="contextcanceled"uncompressedDesc="&{application/vnd.oci.image.layer.v1.tarsha256:8c91d4b467ae2c77ccca2a9caa79d693c4cbeee5b6a7094259e3bf8fe8386796252928[]map[][]<nil>}" WARN[0287]failedtoremovetmpuncompressedlayererror="contextcanceled"uncompressedDesc="&{application/vnd.oci.image.layer.v1.tarsha256:07957ec18cb61c22712e57c4bcea6473c8fbbc06673a53caf796cfa08df324223015680[]map[][]<nil>}" WARN[0287]failedtoremovetmpuncompressedlayererror="contextcanceled"uncompressedDesc="&{application/vnd.oci.image.layer.v1.tarsha256:4b020c57f998fd83880b0d7a9eea02aee587a619362500a7d90249a8f4db1f954096[]map[][]<nil>}" WARN[0287]failedtoremovetmpuncompressedlayererror="contextcanceled"uncompressedDesc="&{application/vnd.oci.image.layer.v1.tarsha256:f05780d37e0f11d63e04fa3c1acae85e9cbb548569eacbb42e58aa8171b70e0771680[]map[][]<nil>}" WARN[0287]failedtoremovetmpuncompressedlayererror="contextcanceled"uncompressedDesc="&{application/vnd.oci.image.layer.v1.tarsha256:22894e74de41edb38ef5c3c0a4d718e3e3481791ef17d9f10e27b5786de2c34f4446208[]map[][]<nil>}" WARN[0287]failedtoremovetmpuncompressedlayererror="contextcanceled"uncompressedDesc="&{application/vnd.oci.image.layer.v1.tarsha256:1738de00ef1d1190bc7e41ac31a9f8f32a1e53cfe4a35ece7dccdc0eb8925a0022995968[]map[][]<nil>}" WARN[0287]failedtoremovetmpuncompressedlayererror="contextcanceled"uncompressedDesc="&{application/vnd.oci.image.layer.v1.tarsha256:7300df2431e2169f8293c5d6e03db7491887d2d93552bf51c4af3150be7ffc8d19662848[]map[][]<nil>}" WARN[0287]failedtoremovetmpuncompressedlayererror="contextcanceled"uncompressedDesc="&{application/vnd.oci.image.layer.v1.tarsha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef1024[]map[][]<nil>}" ctr-remote:refdefault/1/convert-zstdchunked-from-sha256:2c1ce468d9f3d941396801f6e3afc8921466650dd05430fe644cd3537713d27flockedfor1.477800011s(since2024-10-2822:05:52.559014771+0000UTCm=+2109.138592301):unavailable

It's this line of code:

stargz-snapshotter/nativeconverter/zstdchunked/zstdchunked.go

Line 114 in a6b9bdb

log.G(ctx).WithError(err).WithField("uncompressedDesc", uncompressedDesc).Warn("failed to remove tmp uncompressed layer")

I launched ctr-remote with
sudo ctr-remote image optimize --oci \ --net-host \ --period 360 \ --cni \ --gpus "0" \ --zstdchunked \ --zstdchunked-compression-level 8

I tried to bypass the issue somehow but no luck. I have huge ML docker image with bunch of cuda libraries built in as well as several models, so it's a 50 GB of uncompressed ML stuff. Unfortunately, I'm not a golang programmer, so I'm unable to figure it on my own, so just reporting here and hoping that it'd be resolved soon! Would really enjoy having a zstd compression.

[GrigoryEvko]

Putting more context here - looks like the issue with the converter occurs only with images larger than a certain size, and the issue is the size itself: containerd/nerdctl#3623 (comment)