Saner handling/overwriting of autodetected mods

[pjf]

When the CKAN project was first founded, we had the idea of "autodetected" mods that could satisfy dependencies, but not be managed by the client. At the time we had very few mods indexed, and the expectation was that users wouldn't want to uninstall their existing mods just to install them again via the CKAN.

Now things have changed. We have a lot of mods indexed, we have a lot of users, and many folks are exclusively using the CKAN for managing their mods.

We very much need a way to convert "autodetected" mods over to being first class CKAN mods, but this is a relatively hard process. We can't easily detect the version of a mod that the user has installed, and a user may have additional files that the CKAN wouldn't really know about.

For this to work properly, I believe the CKAN would be benefitted by a lot more filesystem awareness, as well as extensions to the spec to make sure we can transition mods correctly. In particular, I believe we need:

• A metadata keyword to indicate files which should be preserved on upgrade.
• A metadata keyword to indicate additional files which should be deleted on upgrade.

These two keywords are useful in their own right, by allowing mods to be upgraded with config intact, or to remove ephemeral files that should disappear during an upgrade. However they also pave the way for sane conversion of pre-installed mods. A migration may consist of:

• Installing the mod as usual, but overwriting files which are in GameData but unowned by CKAN.
• Except where those files are marked as preserved
• In addition, we should delete any files which are specified as needing deletion, provided those files are also unowned.

This means we don't need to know the version of a mod we're coming from, but we can know the version of a mod we're going to.

Once we have a sane upgrade process in place, we can work towards removing autodetected mods entirely. Users with existing installs would be required to convert their mods to CKAN control if they were needed as dependencies.

This is all very much fraught with danger; in particular if one mod's metadata suggests it should delete files owned by another mod (which at least we can detect at install time).

---

Edit: Doing this would also make a fine time to examine the relationship between the Module and InstalledModule classes internally, and if it still makes sense to have them separate.

[pjf]

Adding here for the moment, as closely related:

• We should give users a way to scan for untracked files and report them. These can indicate that we've got incomplete metadata, or simply show the presence of manually installed mods.
• We should provide a repair function for files which returns them to their installed state. This wouldn't touch preserved files by default (as that's dangerous), but can restore regular files. We'd likely have an option to allow dangerous operations (deleting files that should be removed on upgrade, or replacing preserved files with their original versions).
• We might want to give users a way to associate unclaimed failes with particular mods. This functionality probably shouldn't be too accessible, because there are lots of ways user can screw this up.

[boniboni]

I know a lot of people who would benefit immensely from saner overwriting of autodetected mods. It gives so much headaches.

[7ranceaddic7]

How difficult would it be to do a CRC or SHA check on an AD, then compare to the same of a CKAN managed? If same, an option to convert AD to CKAN could be provided? It could reduce a barrier to usage by those who diligently maintain updates.

Also, would it be difficult to just provide a option to complete convert? If a mod is detected as not compatible with 1.0.x, then it is left AD. Perhaps provide this as a CLI option only; that way if someone wants to do it they can do it, but it's all on the user (I'd use this.)

[HebaruSan]

The big challenge in identifying manually installed mods properly is that currently CKAN doesn't know anything about the actual files installed by a mod until it gets downloaded and installed. registry.json contains an installed_files hash that records each file as it's unpacked and copied into GameData, but before that happens, all we have is the hash of the download file (which isn't useful for manually installed mods) plus any install stanzas that are set up for it (which just explain which folders to look at and where to install them). In effect, the roughly 14384 mod ZIP files out there form an extended database for CKAN, of which only a small fraction is captured in our stored metadata.

To be able to look at your GameData and spit out an accurate list of manually installed mods would require extracting all of that info out of the ZIP files and storing it somewhere. It could be a lot of new data to store and download (I'd ballpark it at an additional 58 MB using the below data model, where today's registry.json is 19 MB, so we'd be quadrupling load times and refresh times).

Performance could be challenging. Suppose I see a manually installed file at GameData/ABCD/EFGH.cfg, and I want to figure out which mod it belongs to. Do I search all 14384 mods one by one to see whether they have that file and whether its checksums match? And then repeat that for all of the potentially hundreds of manually installed files? That could take minutes or hours; we'd need some kind of optimized data structure like:

    "installable_files": {
        "GameData/ABCD/EFGH.cfg": {
            "9C793B4F6E3352BF0387BD77CB7BED61150F8FB403D90D1F825ABA379EE3D57D":  [
                {
                    "name":    "MyCoolMod",
                    "version": "v1.2.3"
                },
                ...
            ]
        },
        ...
    },

Then there's the challenge of choosing between multiple available modules that install the same version of the same file (which is why my example above has an array associated with the hash rather than one object). Presumably we would have to try to use clues from the other installed files to determine which one is really installed, but that sounds like one of those complex AI-level tasks to me. In the end we might have to prompt the user to explain which one they really have installed.

Assuming those problems could be solved, there are others. Users might choose to manually install mods that conflict with one another, and the registry would end up in an inconsistent state if they were simply imported as-is. We'd have to write a UI flow to resolve those conflicts.

Seems like a lot of extra work just to let people skip the step of uninstalling mods manually.

[HebaruSan]

A few more observations:

• This would generally be a one-time process; typical users would convert mods from unmanaged to managed when they first start using CKAN and then install with CKAN from that point on
• Probably a minority of users would use such a conversion feature, with many starting from an unmodded install
• Therefore the data is useless 95%+ of the time
• The data would be the same for all users
• The inputs and outputs to the algorithm are much smaller than the data

To me, these factors all point toward centralizing this logic in a web service. The client could generate a JSON summary of what it sees in GameData, send it off to some ksp-ckan.space URL, and receive back the list of mods we think are present. This would remove all the excess data from the client where it's usually not needed, and the response times should still be about the same.

Obviously there are downsides to that as well, such as having to run a web service.

[lamont-granquist]

In my case I just hit this issue through having hit ctrl-c halfway through a ckan scan (i meant ckan search but accidentally typed scan so just ctrl-c'd it) which left a corrupted half-written registry.json file, which crashed ckan doing anything and which was not repairable with ckan repair registry. Blowing it away and then doing a ckan scan restores the list of mods, but now every single one of them are unmanaged and will not update. I actually have in my scrollback the old list of mods with all the versions. I don't need some snazzy list of checksums via a webservice.

This kind of workflow is also necessary when dealing with development, and its painful to go from git installs of stuff like RealismOverhaul, RP-0, etc to wanting to follow CKAN after they are released.

And generally this is wanted by people who have some degree of skill at it, but want an easier workflow than manually deleting all their mods and reinstalling from "scratch". The userbase which is asking for this is probably the kind of userbase who understands what state they're coming from and where they want to get to.

[HebaruSan]

hit ctrl-c halfway through a ckan scan (i meant ckan search but accidentally typed scan so just ctrl-c'd it) which left a corrupted half-written registry.json file, which crashed ckan doing anything and which was not repairable with ckan repair registry.

Could you please file a new issue for this, @lamont-granquist? That sounds like something we could and should fix.

[HebaruSan]

This is showing up in GitHub searches for open issues despite being closed.
Re-opening and re-closing to see if that makes a difference.