New command/option for cache cleanup with customizable retention period

[sigiesec]

Over time, the conan cache grows accumulating stale dependencies. There should be an automatic cleanup mechanism, which automatically removes packages in a customizable way based on some criteria:

• creation/installation time
• time of last use
• channel

For example, we are using testing and stable channels. Packages from testing channels are rebuilt at least daily, so packages older than a day will not normally be used again. Packages from stable channels might be used over months, but could also be removed when they have not been used for some days.

[lasote]

What automatically means exactly? because the conan client is not a daemon, and performing some clean up during an execution will bother the users (as I hate when my OSS makes me wait for software updates in a reboot).
I think it could be a nice parallel project.

[sigiesec]

Maybe as an extra conan gc command, or an option --gc to one or more existing conan commands such as install.

It could also be done via a separate tool, provided the required metadata can be accessed in a defined & stable way.

[danimtb]

I do not like the idea of having things done automatically either. Also that automatic thing would soon start to be customizable with all sorts of criteria: oudated, testing/debugging channels, recipes without packages.... And the conan remove already has some options to filter by settings and so on. For me it is enough.

[sigiesec]

I think this is a must-have when sharing the cache between multiple leaf packages.

And sharing the cache is practically necessary when having a larger number of leaf packages and versions of them. Otherwise the space required is way too large.

Cleaning the cache completely is not an option for performance reasons.

Maybe I am mistaken w.r.t. to the existing -o option but as I understand it it does not serve the use case I described. However, there is no documentation of its behaviour at the moment.

[danimtb]

I agree we should document "outdated from recipe". Here is the issue conan-io/docs#478

[memsharded]

It cannot be automatic, or implicit, it needs to be explicit and configurable by the users, probably in a flexible way.

At the moment, there are no feature that allows to do a conan remove based on other criteria rather than the settings, options, references, etc. No time information.

I wouldn't oppose to a conan remove --policy=old=3months or the like, but I am afraid that it would lead to a myriad of corner cases and different policies, complex to implement and maintain. Policies about "time of last use", might be even more difficult, there is no infrastructure that manages anotating such time, so that would also be necessary. Also, it introduces some asymmetry with the servers, which do not support this feature. Implementing it in all the servers is a very big task, that would require a lot of time.

If anyone is willing to contribute a proof of concept of this feature for the local cache, we could have a look, but at the moment, it is difficult to put it in our roadmap.

[sigiesec]

It cannot be automatic, or implicit, it needs to be explicit and configurable by the users, probably in a flexible way.

Sorry, probably "automatic" was somewhat confusing. I will change the title to "configurable" instead.

I wouldn't oppose to a conan remove --policy=old=3months or the like,

I agree, that would be a good starting point.

but I am afraid that it would lead to a myriad of corner cases and different policies, complex to implement and maintain.

Let's hope it can be kept simple.

Also, it introduces some asymmetry with the servers, which do not support this feature. Implementing it in all the servers is a very big task, that would require a lot of time.

I don't quite understand this point. What does the cleaning of the local cache have to do with the servers?

If anyone is willing to contribute a proof of concept of this feature for the local cache, we could have a look, but at the moment, it is difficult to put it in our roadmap.

Ok, I will check if me or some colleague of mine can do this!

[memsharded]

I don't quite understand this point. What does the cleaning of the local cache have to do with the servers?

The conan remove command works basically the same for the local and remote packages. The command is the same, and it shares most of the arguments, and also the internal code. You can remove from remotes with conan remove ... -r=myremote. The new policy argument will only be valid for local cache, but it is a matter of time that someone request it to work for servers too, and that is a much bigger feature, I don't know if it is even possible, as Artifactory as its own way to do that (I think the JFrog-cli can do these kind of operations).

Ok, I will check if me or some colleague of mine can do this!

Excellent, if finally going to give it a try, please try to keep it as "minimal" as possible, submit it as soon as possible to gather early feedback, instead of working for a long time. Many thanks!

[jgsogo]

I would propose a different approach, more explicit on the user side: what if Conan just provides a list with all the packages in the local cache (and maybe accept a matching pattern like the search command does)?. Then, the user can iterate this list and remove the packages with the existing command, or compile all of them using a new profile,...

In the future, this list command, can also add some metainformation like creation or download date, last used (hard to know),... and maybe we can provide extra arguments to filter by this meta-information, or by settings. I think it is safer to make a mistake in a conan list than in a conan remove

Other servers can implement this endpoint or not, an let the user know the full list of packages; but they more likely to do so than to provide a batch remove command with wildcards.

[thbeu]

Are there any nears plans to provide some conan cache cleanup options at all? Thank you.

.conan\	69,86 G	07.11.22 14:33	----

[memsharded]

Yes, Conan 2.0 has provided some infrastructure for storing the LRU date of packages, and that will be used for implementing such behavior. It is planned as a 2.X feature, that means, after stabilizing 2.0, will be in the backlog.

[memsharded]

Conan has finally implemented this in #14054, it has been merged it will be in next 2.0.14 release!