Provide mechanism or documentation for cache cleanup

[bizrad]

What would you like Renovate to be able to do?

It would be nice to offer a method of pruning the cache. On a run with and empty cache directory it populates 32M of space in our instance and after 6 months of hourly runs this grows to 88G.

This is also an issue when using GitLab cache which zips -> uploads and downloads -> unzips the cache on each run, so over time runs begin to take longer and longer.

The current workaround it to completely wipe the cache on a regular basis or when a disk fills up.

Alternatively, if this is not implemented as a feature in code then documentation should be updated provide information on how cache can be pruned by running other commands like find /renovate/cache -type f -mtime +30 -delete or if this is not possible and the whole cache should be wiped regularly.

If you have any ideas on how this should be implemented, please tell us here.

I think it would be preferable to have a config option that can enable cache cleanup (true/false).
A second option to tune age of cache items may also be desirable (e.g. 30d)

Is this a feature you are interested in implementing yourself?

No

[viceice]

I think we can manage some cleanup in cache teardown, which is called anyways.

Redis a s doing this automatically, so i think we can do something similar in renovate file / cacache.

[viceice]

Maybe simply run verify at end?

https://github.com/npm/cacache#verify

[rarkins]

Sounds like verify would clean up some cacache internals, but I don't think it would help us clean up our own expired cache entries.

[rarkins]

I think what's needed is to run ls to get every entry, then delete it if it's expired. We may need to try their streaming approach to not run out of memory when it's large. This should be done at the very end of a run

[buffcode]

Maybe Symfony's logic can be reused here:

• prune() scans the local cache directory using a generator ¹
• all cache files have the expiry time in the first line ^{2 3}, so not the whole cache file has to be retrieved / parsed

Footnotes

1. https://github.com/symfony/symfony/blob/6.1/src/Symfony/Component/Cache/Traits/FilesystemTrait.php#L34 ↩

2. https://github.com/symfony/symfony/blob/6.1/src/Symfony/Component/Cache/Traits/FilesystemTrait.php#L98 ↩

3. https://github.com/symfony/symfony/blob/6.1/src/Symfony/Component/Cache/Traits/FilesystemTrait.php#L39 ↩

[viceice]

I think the ls.stream function is the way to go.

[b-a-t]

Is there any development happening on this feature request? We are forced to periodically stop the RenovateBot(-on-premise) and completely wipe out the cache, as it grows uncontrollably and takes all the given disk space(OK, we allocated it a separate LV, so it won't bring the whole machine down).

[rarkins]

No development planned any time soon, but community PRs would be welcome.

What I suggest:

• Add a new global option cleanCache which defaults to never but has options first, and last
• If option is first or last (i.e. at beginning or end) then use the streaming ls command to fetch all values and delete them if expired

[mrsimo]

Just wanted to mention that since recent changes we've started noticing a growth in cache use:

Went from almost stable use to linear growth. That started when we upgraded from 34.77.1 to 34.94.0 (we do weekly upgrades). The changes that stood out to me were #19623 and #19059, but I've no idea. All the growth is in cache/renovate/renovate-cache-v1/content-v2/sha512.

We'll look into clearing the cache manually somehow, but wanted to leave this message to bring up that recent changes might make this problem worse sooner for folks.

[viceice]

I'm using redis cache for that reason 😉

[renovate-release]

🎉 This issue has been resolved in version 35.0.0 🎉

The release is available on:

• GitHub release
• 35.0.0

Your semantic-release bot 📦🚀