Do something about cache housekeeping?

[pfmoore]

What's the problem this feature will solve?
Pip stores data in its cache, but never clears out obsolete data.

Describe the solution you'd like
Some means (automated or manual) for no-longer-needed cache entries to be cleared out.

Alternative Solutions
It's possible to just delete the cache altogether, as there is nothing in there that won't be recreated as needed, but this is an "all or nothing" solution.

Additional context
Prompted by the discussion here

[chrahunt]

Related to #3138.

[pradyunsg]

Also related to #4685, since that could be how you perform this.

[gutsytechster]

Is this still relevant as we finally have a pip cache command with some useful options like purge, remove etc?

[duckinator]

I think, given pip cache as it is and #7372 (having it handle more than wheels), I think the main thing left for this issue is the possibility of having automatic cache cleanup. Is that correct, @pfmoore, or do you have other ideas?

[pfmoore]

Agreed, and as discussed in #8474, I'm ambivalent at best over the idea of an automatic cleanup. So I'm going to close this as complete, and leave automatic cleanup as something for someone else to raise if they feel like it.

Thanks for your work in making this happen @duckinator!

[pfmoore]

Reopening, as I realised via this discussion that this issue was originally triggered by the question of tidying up outdated selfcheck files.

IMO we still need an automated solution for clearing up obsolete selfcheck files. I'd expect that to be something along the lines of whenever we do a selfcheck, we check all the other selfcheck files and delete any that refer to directories that no longer exist. I don't think it should be down to the user to run a purge, nor do I think we should leave files for non-existent environments indefinitely.

[pfmoore]

Bah. Long day. Thanks for letting me know!

[itamarst]

#2984 is going to make this worse, by potentially have two copies of the same file in the cache.

Here is a proposal for a solution:

No more than once a day (to prevent performance impacts), after installing a package (so cache information is accurate), clear old entries from the cache.

Two potential strategies for deciding which entries to delete and how many:

1. Date-based: Find all cached entries with access time (atime on Unix) more than 90 days in the past, delete them.
2. Size-based: Try to keep the cache below e.g. 5GB. Delete entries until that size is reached, from oldest access time to newest.

Given data science packages can be huge, and that disk space is cheap, I would suggest that date-based caching is probably better.

[pooh22]

As sysadmin at a university, we get mails on a regular basis from students who can't figure out why they get over quota mails. Most of the time it's their pip cache filling up the initial 5G of storage they get. It would be very nice if pip would respect a setting (global) that maximises their cache size. It should be checked every time the pip command runs and enforced either automatically or via a clear suggestion to the user. The all or nothing option of purge seems like a poor way of cache management.