Option to clone smaller instances with junction points (Windows) or symbolic links (Unix)

[HebaruSan]

Background

It's common for users to make extra copies of KSP1 so they can install different mod lists in each one. CKAN supports this by tracking multiple instances and providing a clone operation, which copies an existing instance to a new folder and adds it to CKAN.

Motivation

• Each game instance contains about 5.5 GiB of files that come with the game and are necessary for it to run. Copying these takes a long time, and having multiple copies of them wastes a lot of space.
• Each game instance also contains files generated during gameplay and while CKAN is operating that are relevant to that instance but not really to clones. For example, screenshots are relevant to the instance where they're created, but having multiple copies of them isn't generally desirable. Copying these into cloned instances also wastes some time and space.

Changes

• Now there is an option to share the stock files between the clone and the original instance, either via ckan instance clone --share-stock or a new "Share stock files" checkbox in GUI:
  
  which is done by making junction points (Windows) or symbolic links (Unix) for the following directories instead of copying them:
  • KSP_x64_Data (the main stock game contents folder)
  • KSP_Data (same but on Linux)
  • GameData/Squad
  • GameData/SquadExpansion
  • Some stuff from the now defunct launchers, just in case those are present
• Any folders that are already symbolic links (e.g., if you're cloning a previous thin clone or if you're a mod developer with symbolic links from GameData to your source folders) will be duplicated in the cloned instance as symbolic links to the same targets
• Now the following directories' contents aren't copied into cloned instances of KSP1; the directories are created, but left empty:
  • CKAN/history (old mod lists)
  • CKAN/downloads (deprecated but might still contain files in very old instances)
  • saves
  • Screenshots
  • thumbs
  • Missions
  • Logs
• Equivalent updates are made for KSP2 instances

This will allow the creation of game instances that are much smaller but still playable (a thin clone of my Steam instance shrinks from 7 GiB to 81 MiB, or about 1% of its original size, and is very quick to generate).

Note that the CKAN registry and the currently installed mods will still be copied across, so users will still be able to set up some mods in one instance and then clone them elsewhere as a starting point for a different mod list.

Fixes #4087.

[HebaruSan]

Hey @ProgrammerFailure, this is the pull request for cloning with symlinks. You can click the Checks tab, then the Artifacts dropdown, to download a test build with the "Share stock files" checkbox in the cloning window:

I've only tested it on Linux so far, so it needs to go for a test drive on Windows before merge. If you have a chance, please try it out and let us know if you see any issues. Thanks!

[ProgrammerFailure]

Hey, this didn't actually create the symlinks for me. As expected, stock files weren't copied, but symlinks also weren't created. Tested with three preexisting CKAN instances, one mod-free, one dev install, and one heavily modded game. No symlinks for Squad were created.

I used the CKAN you provided, clicked share stock files, and cloned an instance to no effect. I also had to disable the windows batac warning.

[ProgrammerFailure]

Windows 11, if that helps

[ProgrammerFailure]

No problem, still looking forward to it.

[ProgrammerFailure]

I can confirm it works with asroot. CKAN could request admin or just make a bat file and tell the user to run it?

[ProgrammerFailure]

For what its worth, I'm fine with granting CKAN admin once per clone operation, and considering it saves 5-6 GB, I imagine others will be too.

[ProgrammerFailure]

True. Plus, I suppose there are still people without admin access to their system? What do you mean by discoverability?

[JonnyOThan]

CKAN actually tries very hard to make sure that it’s NOT running as admin. This causes countless problems with users who run elevated all the time. No one understands this stuff.

There’s gotta be a way you can request UAC elevation? I’m kinda surprised that CKAN wouldn’t have that permission level considering that it just created the directories, right? Maybe there’s some option for setting the permission when creating. Or can the clone be created elsewhere and then moved into position?

[JonnyOThan]

Oh interesting, you have to start a new process: https://stackoverflow.com/a/573150

Maybe just run console ckan with all the parameters and —as root?

[ProgrammerFailure]

No, I am not. Should I test with it enabled?

The article mentions developing with VS. I am doing this, but it's just a plugin, not an app.

[JonnyOThan]

Just realized something: I never actually use symlinks on windows when setting up references; I use junctions (mklink /j). I've never had to switch to an admin account or elevate privileges. I'm not exactly sure how to do that programmatically but maybe we can just invoke mklink directly on windows?

https://stackoverflow.com/questions/3615129/how-to-create-a-junction-point-using-winapi

[HebaruSan]

@JonnyOThan, thanks for the idea! That does indeed look like the better approach on Windows. We're only making symlinks to the absolute paths of directories, so the only difference for us would be the security stuff.

[JonnyOThan]

I'm like 90% sure they can...I recall doing that at a previous job.

[techman83]

@HebaruSan yes they do. It's how I used to point some games on a different drive when stream didn't support install locations. I've probably still got the PowerShell script for the task somewhere.

https://en.m.wikipedia.org/wiki/NTFS_links#:~:text=While%20NTFS%20junction%20points%20support,support%20for%20cross%2Dfilesystem%20links.

[HebaruSan]

I think the problems I was having were caused by thinking that strings consist of 1-byte chars like in C, but they're actually 2-byte chars (the API requires byte counts to be specified in several places).
Latest push:

• Junctions instead of symlinks on Windows
• The checkbox is always visible and checked by default

Should be a lot more useful now that all that security craziness is gone. I'll probably merge this version unless I or anyone else finds an issue shortly.

[ProgrammerFailure]

@HebaruSan ,

I wanted to download this but it says that these artifacts have expired. Is there any way to access it without building CKAN myself?

[HebaruSan]

@ProgrammerFailure yes, you can try the dev build here:
https://ksp-ckan.s3-us-west-2.amazonaws.com/ckan.exe

[ProgrammerFailure]

Thanks! Will that link always go to the current dev build?

[HebaruSan]

Yup, it gets built and replaced every time we merge something to the master branch.

[ProgrammerFailure]

On trying to make a symlinked instance, I get this error:

CKAN.InvalidKSPInstanceKraken: Exception of type 'CKAN.InvalidKSPInstanceKraken' was thrown.
   at CKAN.GameInstanceManager.SetCurrentInstance(String name)
   at CKAN.GUI.CloneGameInstanceDialog.<buttonOK_Click>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

And it doesn't make the instance. Did I mess up something? Reputation based protection is disabled.

[HebaruSan]

Did I mess up something

Nope, an unrelated bug caused by a later change is causing the "copy from" instance's path to not fill in to this box:

I just pushed a fix for this, which our GitHub workflows will process and eventually generate a new dev build, which the previous dev build should prompt you to install when you launch once it's ready.

Thanks for catching that!

[ProgrammerFailure]

I'm now getting this error:

CKAN.InvalidKSPInstanceKraken: Exception of type 'CKAN.InvalidKSPInstanceKraken' was thrown.
   at CKAN.GameInstanceManager.SetCurrentInstance(String name)
   at CKAN.GUI.CloneGameInstanceDialog.<buttonOK_Click>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

[ProgrammerFailure]

Latest dev version, and the box does get autofilled
I turned off real-time protection and it didn't have any effect.

[HebaruSan]

Thanks again, found and fixed another recent issue with that popup.
The next dev build you try should work.