xrepo: a remove flag which automatically removes older versions of packages

[SirLynix]

Is your feature request related to a problem? Please describe.

I'm working with some packages that get updated pretty often, and only use the last version of those.

However, when I'm updating, previous packages versions stay there taking up space, I'd like a way to remove these older versions.

Describe the solution you'd like

Maybe a --only-outdated, like xrepo remove --only-outdated --all <package> flag which would select only package that got a more recent version installed (with the same config).

Describe alternatives you've considered

Removing all packages before updating, a bit cumbersome.

Additional context

No response

[waruqi]

try xrepo clean --all package, it will remove all unused packages.

[SirLynix]

I didn't know about xrepo clean, however it doesn't seem to work for all packages.

For example, nazarautils:

C:\Users\lynix>xrepo scan nazarautils
scanning packages ..
nazarautils-2023.04.11:
  -> 3a8ef2be90c044599cb99c00c15cd73c: windows, x64
    -> {debug=false,shared=false}
nazarautils-2023.04.13:
  -> 30baa9d054ca4797b9507ad1b4a18c11: windows, x64
    -> {debug=false,shared=false}
nazarautils-2023.04.23:
  -> b610f4dade3d444b9c8893253545f85f: windows, x64
    -> {debug=false,shared=false}
C:\Users\lynix>xrepo clean nazarautils
clearing packages ..
clearing caches ..

when I look at the references.txt file from the 2023.04.11 version I get this:

{
    ["C:\\Projets\\Perso\\NazaraNext\\NazaraEngine"] = "230414",
    ["C:\\Users\\lynix\\AppData\\Local\\.xmake\\cache\\packages\\2304\\n\\nzsl\\2023.03.29\\source\\nzsl"] = "230414"
}

nzsl 2023.03.29 is an older version as well, and it was referenced from its cache?

[waruqi]

right, it's from cache.

[waruqi]

I improved it, it will remove cache first.

[SirLynix]

It removed a few more packages, but nazarautils 2023.04.11 is still there with the same references.txt

{
    ["C:\\Projets\\Perso\\NazaraNext\\NazaraEngine"] = "230414",
    ["C:\\Users\\lynix\\AppData\\Local\\.xmake\\cache\\packages\\2304\\n\\nzsl\\2023.03.29\\source\\nzsl"] = "230414"
}

C:\Projets\Perso\NazaraNext\NazaraEngine>xmake require --info nazarautils
The package info of project:
    require(nazarautils):
      -> description: Header-only utility library for Nazara projects
      -> version: 2023.04.23
      -> license: MIT
      -> urls:
         -> https://github.com/NazaraEngine/NazaraUtils.git
            -> 8f7d5f554cb53d1a9bb4ef56801d209e7c39b661
      -> repo: nazara-engine-repo https://github.com/NazaraEngine/xmake-repo
      -> cachedir: C:\Users\lynix\AppData\Local\.xmake\cache\packages\2304\n\nazarautils\2023.04.23
      -> installdir: C:\Users\lynix\AppData\Local\.xmake\packages\n\nazarautils\2023.04.23\b610f4dade3d444b9c8893253545f85f
      -> searchdirs:
      -> searchnames:
         -> nazarautils (git)
         -> NazaraUtils (git)
      -> fetchinfo: 2023.04.23
          -> sysincludedirs: C:\Users\lynix\AppData\Local\.xmake\packages\n\nazarautils\2023.04.23\b610f4dade3d444b9c8893253545f85f\include
          -> version: 2023.04.23
          -> license: MIT
      -> platforms: all
      -> requires:
         -> plat: windows
         -> arch: x64
         -> configs:
            -> shared: false
            -> debug: false
      -> configs:
      -> configs (builtin):
         -> debug: Enable debug symbols. (default: false)
         -> shared: Build shared library. (default: false)
         -> pic: Enable the position independent code. (default: true)
         -> lto: Enable the link-time build optimization. (type: boolean)
         -> vs_runtime: Set vs compiler runtime.
            -> values: {"MT","MTd","MD","MDd"}
         -> toolchains: Set package toolchains only for cross-compilation.
         -> cflags: Set the C compiler flags.
         -> cxflags: Set the C/C++ compiler flags.
         -> cxxflags: Set the C++ compiler flags.
         -> asflags: Set the assembler flags.
      -> references:
         -> 230424: C:\Projets\Perso\NazaraNext\NazaraEngine
         -> 230423: C:\Users\lynix\AppData\Local\.xmake\cache\packages\2304\n\nzsl\2023.04.13\source\nzsl (not found)

[waruqi]

  -> references:
    -> 230424: C:\Projets\Perso\NazaraNext\NazaraEngine

Because there is another valid project that is referencing it.

you can move NazaraEngine to other position.

xrepo clean does not guarantee complete removal of all unused packages, it may not detect reference relations accurately, if you want to remove a package reliably, use xrepo remove --all xxx to force to remove it.

xrepo clean just simply cleans up as many unused packages as it thinks are safe to delete.

[SirLynix]

Yes, that's the issue, NazaraEngine is not using this package but an updated version of it.

xrepo clean does not guarantee complete removal of all unused packages, it may not detect reference relations accurately, if you want to remove a package reliably, use xrepo remove --all xxx to force to remove it.

I understand, perhaps a flag to remove outdated version of a lib may be useful in this case?

[waruqi]

no, xmake does not know quickly, precisely and simply what is outdated

At least xrepo clean can quickly remove most of the outdated packages, although it is not entirely accurate, it is sufficient.

If you want to remove it with complete precision, you can use xrepo remove.

[SirLynix]

By outdated I meant "where a newer version was installed", that is "same config but more recent version".

However I didn't know about xrepo clean, so now I think it's more about the references.txt file not being updated after a package update?

Would it be possible for xrepo clean reference checker to be improved to deal with package updates?

[waruqi]

When installing a package, xmake can only fetch the reference.txt path of the current new version of the package and update it, it cannot fetch the previous package instance and reference.txt

[SirLynix]

I understand that, but maybe the reference checker could check that the repository uses that version of the package (from xmake cache)?

[waruqi]

Accessing the cache of other projects is a little hacking and may have compatibility issues.

[SirLynix]

I understand. Maybe xmake config could store a file dedicated to this? Like .xmake/windows/references.txt which would store all packages and their hashes with the sole purpose to let xrepo clean know if the project uses that package. This would allow xrepo clean to be more precise and work with package updates/config change.

and for backward compatibility, xrepo clean could have the same behavior as today if that file doesn't exist.
What do you think?

[waruqi]

I will consider it later.

[waruqi]

#3689

xmake update -s github:xmake-io/xmake#ref

[SirLynix]

It looks like it's working great! Thank you!

[SirLynix]

Oh, I just noticed it worked only for the current mode if your packages depends on this. I know this won't be exhaustive but could it be possible to store references per mode?

[waruqi]

No, that's all it can do at the moment. Not only mode, but also various configs, custom options, platforms, and arch may affect package hash. As I said before, it can't be completely accurate.

[SirLynix]

Yes I know it wouldn't be accurate, but if we put configs to the side and keep plat+arch+mode I think it would work for most cases.

A bit like what package lock file does

[waruqi]

require lock now also only deals with host/arch, not mode.

Even though I now support host/arch/mode, perhaps there may be feedback from other users that more configurations need to be saved. There is no set standard for this.

I think this is a bit overly complicated, and it's enough to just deal with the current configuration. Also the current project does not use any other configured packages. I don't intend to consider handling other configurations.

[SirLynix]

I understand your point of view, but I think this will be a very common issue on Windows, all of my packages have different hash in debug and release mode, as it changes the vs_runtime (I use "MDd" in debug and "MD" in release, by the way I think this should be the default for mode.debug and mode.release as MT runtime have some issues, as I explained here: #3683 (comment)).

I find it quite normal that having different configs changes the references.txt and that xmake doesn't handle this, but for arch/config that's another issue, because when you run xmake project -k vsxmake and it installs packages for debug/release and x86/x64, you don't expect xrepo clean to remove most of the newly installed packages.

[waruqi]

I understand your point of view, but I think this will be a very common issue on Windows, all of my packages have different hash in debug and release mode, as it changes the vs_runtime (I use "MDd" in debug and "MD" in release, by the way I think this should be the default for mode.debug and mode.release as MT runtime have some issues, as I explained here: #3683 (comment)).

I wouldn't modify it, it would break a lot of existing user projects. mode.debug is just a built-in rule, you can do not use it. Then customise debug mode to configure it.

I find it quite normal that having different configs changes the references.txt and that xmake doesn't handle this, but for arch/config that's another issue, because when you run xmake project -k vsxmake and it installs packages for debug/release and x86/x64, you don't expect xrepo clean to remove most of the newly installed packages.

But it is also possible that the user no longer compiles using other modes and arch, then these packages can never be cleaned up. They may also feedback a similar problem, just as you did when you reported that the entire project's packages were not cleaned up.

Also xrepo clean will prompt you if you want to remove it, and you can choose not to remove packages from other modes.

So no matter how it is done, it is imprecise and cannot fully satisfy the needs of all users. I think the revert patch is probably the right one.

If you want to accurately clean up the current project of outdated packages, you can only xrepo remove --all xxx and reinstall the new version.

[waruqi]

Also, since xmake project -k vsxmake is executed in a single process, there is a way to solve this problem by merging all the mode/arch packages together and writing them to references.txt. Simply using a global memory cache is all that is needed.

I have improved it.

[SirLynix]

This is good enough for what I do, thank you!