Pins with expiry times

[matthewrobertbell]

Version information: 0.4.11

Type: Feature / enhancement

Severity:

Description:

A few times I have wished that I could pin stuff (using the API, from a script that runs regularly), but only for a certain amount of time. This is useful for data which becomes less important after a certain amount of time to me. Without this feature, the client has to track pins in a database then remember to unpin over time.

It could potentially work like this:

ipfs pin $hash --hours=72

or

ipfs pin $hash --days=30

etc.

[whyrusleeping]

ipfs pin $hash --duration=40m|16h|30d could work. Though we would need to entirely rework how pinsets are stored and maintained on disk. Right now its a set, we would need to turn it into a map for this.

[djdv]

@whyrusleeping
Would it make sense to keep that map separate, as a different set that consists of [hashes:times], which is checked at some interval? Maybe tie it to the same mechanism that the GC timer works on?
Given that these are inherently temporary rules, it seems less integral to store them alongside the pins in a set if they're just to be removed later.
That being said, I have no idea if that would be any less complex or worth doing compared to reworking pinsets as a whole and just keeping it all together.

[whyrusleeping]

@djdv Hard to say without looking more deeply into this. But keeping the information separate might be a workable solution, it would definitely make iterating over the timed pins to do gc much easier.

[matthewrobertbell]

It'd also make sense to have "ipfs pin ls" show the time remaining, and maybe the time it was originally pinned for as well.

Would running the pin command a second time for a hash reset the duration?

[ghost]

I'm not sure it's worth it to work this into go-ipfs. The cost of reworking all the pinning-related code is very very high and it's arguably the most sensible part of the codebase because mistakes can lead to data loss.

Would it work to build this on top of IPFS? It's as simple as calling ipfs pin rm <hash> once time is up.

[matthewrobertbell]

The reason I suggested that it be built into go-ipfs is that it seems like a common pattern which would be most reliable if implemented once in go-ipfs.

Could this possibly be implemented as a layer on top of the existing pinning code (within the same binary), internally using the ipfs pin API? Not familiar with the internals, so not sure if that would make sense.

[Stebalien]

Personally, I'd kind of like to do away with "pinning" in general as it has some significant usability and scalability problems:

1. No concept of "ownership" (which application owns what pin).
2. No names (have to manage separately).

I'd like to just replace "pins" with a filesystem for files and a key/value store for DAG objects. This way, everything that's named is pinned and objects can have multiple names/pins. The tricky part here would be separating recursive and non-recursive pins (and all pins in-between) but, IMO, this is something that could be encoded within IPFS itself. That is, IPFS directory entries would include a pinning policy (recursive, directories only, delegate) for the linked file/directory. However, this would be an even larger undertaking (and probably won't happen for a while).

I bring this up because I think we should avoid putting too much effort into improving the current system (beyond simply optimizing it).

---

This wouldn't cover your direct use-case but, with this system, you could easily just name links to files based on their expiration times and GC them using an external system. Regardless of what we do, we will need to allow for external GCs as we can't account for all possible configurations in go-ipfs itself.

[Ericson2314]

I left a comment #4559 (comment) that probably should have gone in this thread instead. Basically, I think I agree with the rough take of @Stebalien's comment above: pinning needs a concept of ownership, and temporary pinning is better served by some RAII thing than timeouts: you want the client to open something like a transaction which when closed will tell the server it's OK to unpin.