[feature] Track artifacts in Conan cache, expire least recently used

[w3sip]

One of the challenges we see is maintaining free space on CI nodes, as well as development machines, with Conan cache on them. There is, of course, an atomic option to blow it all away, but that means the next few (or not so few) builds will be slow, retrieving the dependencies from the remote.
Something that would help is tracking access to the artifacts, and ability to remove least recently used ones. A few strategies I could see are:

• shrink the cache to a maximum size specified, deleting LRU items

• delete all items that hasn't been accessed in X days

• same as above, but with a search constraint

• I've read the CONTRIBUTING guide.

[memsharded]

Hi @w3sip

This has been requested before, e.g:

• How to automatically expire a package? #1784
• Implement something like git gc to clean out older versions of dependencies. #5453

I suggest taking it into account for Conan 2.0, as it seems that we will be redesigning at least parts of the cache, and storing the access time metadata shouldn't be difficult.

Exploiting that data in conan remove for time-based comparisons, doesn't seem very difficult, but we should also design the CLI 2.0 accordingly, so it is consistent.

[memsharded]

We have added some infrastructure to the Conan 2.0 new cache, with provision for LRU timestamps that can be leveraged.
But it has not been complete, and it is not in scope for 2.0 GA, so moving it to 2.X roadmap

[robindegen]

This would be incredibly helpful for our infra. We now have to manually track cleanups since over time build servers and docker cache (volumes) just fill up.

[memsharded]

Hi @robindegen

You want to track this PR, this is the one proposing an implementation: #14054

[robindegen]

Hi @robindegen

You want to track this PR, this is the one proposing an implementation: #14054

Thank you. I will track that one.

On a sidenote; I completed the migration of part of our infra to Conan 2.0 and was pleasantly surprised. Its unfortunate that the api had to be broken to do the migration (which I now understand more why) but Conan 2 is genuinly better and has been very pleasant to work with so far.

[memsharded]

On a sidenote; I completed the migration of part of our infra to Conan 2.0 and was pleasantly surprised. Its unfortunate that the api had to be broken to do the migration (which I now understand more why) but Conan 2 is genuinly better and has been very pleasant to work with so far.

Thanks very much for your kind words! That makes us super happy 😄

We know that breaking changes are always inconvenient and require an extra effort from users, but we really thought that these were necessary to take Conan to the next level. And we are very excited, not only because the current 2.0 release is already much better, but also because its better architecture and design will allow to work on many new improvements and features, like the recent "pkg-lists" that landed in 2.0.7, but also packages metadata (getting ready for 2.0.8), and many other great enhancements.

[memsharded]

Conan has finally implemented this in #14054, it has been merged it will be in next 2.0.14 release!