Introduction of packages directory and how mod-managers should handle it

[GeckoEidechse]

Making this issue to discuss #503 with mod-manager authors. Mentioned PR overhauls the mod loading process by allowing to read Thunderstore packages natively.
For reference, Thunderstore packages are just the zip downloaded from Thunderstore which in turn contain Northstar mods.

What's changing and why?

The idea behind #503 is to basically simplify mod installation by separating core mods from user installed mods and reading Thunderstore packages directly. This is done in an effort to overhaul the mod system which is currently being discussed in R2Northstar/Northstar#472

For mod-managers this also massively simplifies the installation process as you no longer have to extract the Northstar mods from a Thunderstore package but rather just extract the zip into a folder of the same name in R2Northstar/packages.

So if you have a zip from Thunderstore called EladNLG-HUDRevamp-2.0.0.zip you just extract all the contents into a directory EladNLG-HUDRevamp-2.0.0 located in R2Northstar/packages.

This also makes updating mods and replacing old files trivial, as you can just download the new version and extract it into the packages directory just like a fresh install, simply and deleting the previous version if the update was successful.

And finally that means that the whole deal with copying manifest.json into each Northstar mod and creating thunderstore_author.txt is no longer necessary, as the manifest is already contained in the Thunderstore package and the Thunderstore author name can be deduced from the folder name (in fact the folder name matches the entire Thunderstore mod string).

Here be dragons

The "caveat" in this simplified approach is that we are also require the mod to follow the specification exactly as based on past experience allowing for any edge case handling etc just causes more issues in the long run.

Specification

This means, if you do extract the zip contents to a folder that does not follow AUTHOR-MOD-MAJOR.MINOR.PATCH, it will be ignored by Northstar launcher.

Similarly as we are keeping the previous folder structure in tact to not break mods on Thunderstore right now, the Northstar mods will have to still be located in a subfolder called mods. Mods that do not adhere to this setup will be ignored by NorthstarLauncher.

Backwards compatibility and broken mods

Note that the structure within the Thunderstore package is unchanged for this update. This means that it's still a folder that contains a mods folder that then contains Northstar mods.

As there are some broken mods on Thunderstore right now that fail to adhere to this setup (and are therefore not installable by most mod-managers under the current install method), so you wanna make sure to check whether a mod is correctly formatted instead of just extracting the zip without any checks.
Whether you just check whether a mod is correctly formatted and fail out if it isn't or whether you attempt to fix it for the user is up to you.
(Although I do recommend just to error out to avoid accidentally introducing more issues when you perform error handling)

What would I need to do to make my mod-manager compatible?

• Support reading packages folder (otherwise you might not be able to see all installed mods)
• Support installing into packages folder instead of mods based on new specification (otherwise you might annoy other mod-managers that have to maintain legacy format because of you)
• Add some function to convert existing structure to new one (not strictly necessary but that means you can start relying on packages as opposed to mods faster 👀

What stays the same?

• enabledmods.json
• Structure of packages on Thunderstore
• Mods that can still be located in mods/
• ???
• Whatever else hasn't been said to change above

Timeframe:

The rough timeline of stuff (without actual information of when what happens)

1. Discussing change (this issue here)
2. PR merged
3. Mod-managers work on preliminary support
4. Version of Northstar with packages released
5. Mod-managers fully support new setup
6. Documentation in wiki is updated
7. ???
8. Profit
9. Further changes to mod-system once this change has propagated

TL;DR:

We go from

R2Northstar/
└── mods/
    ├── S2.GlitchAutoLighting <---------- 3rd party mod installed by user manually or mod-manager
    │   ├── manifest.json     <---------- Thunderstore manifest added by mod-manager
    │   ├── mod/       <----------------- contains Squirrel scripts that are part of mod
    │   ├── mod.json   <----------------- defines mod for Northstar
    │   └── thunderstore_author.txt  <--- added by mod-manager
    │
    ├── S2.GlitchSkybox
    │   ├── manifest.json
    │   ├── mod/
    │   ├── mod.json
    │   └── thunderstore_author.txt
    │
    ├── Northstar.Client/
    ├── Northstar.Custom/
    └── Northstar.CustomServers/

to

R2Northstar/
├── mods/
│   ├── Northstar.Client/
│   ├── Northstar.Custom/
│   └── Northstar.CustomServers/
│
│
└── packages/  <--------------------- this is new
    └── S2Mods-GlitchOverhaul-1.2.11 <--- whole Thunderstore package with mods contained within
        ├── manifest.json    <----------- Thunderstore manifest, extracted directly from the zip like other files
        ├── mods/
        │   ├── S2.GlitchAutoLighting
        │   │   ├── mod/
        │   │   └── mod.json
        │   │
        │   └── S2.GlitchSkybox
        │       ├── mod/
        │       └── mod.json
        │
        └── README.md  <----------------- some residue file from the Thunderstore package, we can just leave this in there

---

Footnotes:

R2Northstar/packages refers to PROFILE_FOLDER/packages. R2Northstar is simply the default profile name, hence I picked it here ^^

[GeckoEidechse]

Oh boi this got way longer of a read than I planned.

Anyway paging

• @0neGal for Viper
• @BigSpice for VTOL
• @AnActualEmerald for libthermite and papa
• @GeckoEidechse for FlightCore (hey that's me :D)

[ASpoonPlaysGames]

A neat advantage of this is that things like the icon.png and stuff will be in the package folder, so the mod manager can show the icons in the installed mods browser and stuff

[0neGal]

What about plugins? Besides a security warning when installing, do they get scanned for and loaded when inside <profile>/packages/AUTHOR-MOD-MAJOR.MINOR.PATCH/plugins/? And thereby have no actual edge cases to worry about...?

[F1F7Y]

What about plugins? Besides a security warning when installing, do they get scanned for and loaded when inside <profile>/packages/AUTHOR-MOD-MAJOR.MINOR.PATCH/plugins/? And thereby have no actual edge cases to worry about...?

They do not, <profile>/packages is only scanned for mods

[GeckoEidechse]

What about plugins? Besides a security warning when installing, do they get scanned for and loaded when inside <profile>/packages/AUTHOR-MOD-MAJOR.MINOR.PATCH/plugins/? And thereby have no actual edge cases to worry about...?

Congrats, you're the first person to notice that I conveniently left out plugins from the mentioned PR :,)
I actually haven't implemented that yet in the launcher PR, will probably do it in a separate one given that plugins are not used a lot.

But yeah, plugins would be at <profile>/packages/AUTHOR-MOD-MAJOR.MINOR.PATCH/plugins/ the same way Squirrel mods are at <profile>/packages/AUTHOR-MOD-MAJOR.MINOR.PATCH/mods/, so no edge case to worry about.

Functionality should arrive in launcher in time so do NOT put them in <profile>/plugins/ when rewriting mod install/loading logic.

[GeckoEidechse]

Probably should also poke @pg9182 cause we probably wanna add support to the northstar-dedicated Docker image as well ^^

(Or at least add docs / update sample yaml, not sure if the container does anything special in advance to check stuff in regards to mounted mods)

[0neGal]

Functionality should arrive in launcher in time so do NOT put them in <profile>/plugins/ when rewriting mod install/loading logic.

Thank you for that, I was considering rewriting the entire mod install part of Viper, because it's very messy right now, and it'd then also add support for plugins, but I've made very little progress on it due to personal reasons. But this makes it much easier

Even if it's not implemented on release with packages/ it'll likely be fine, there's a reason it's been a low priority for Viper to begin with.

[AnActualEmerald]

libthermite basically supports this now on the dev branch, this change actually makes my job so much easier 🫡

relevant example: https://github.com/AnActualEmerald/thermite/blob/dev/examples/packages.rs

[F1F7Y]

Probably should also poke @pg9182 cause we probably wanna add support to the northstar-dedicated Docker image as well ^^

I dont think this is a blocking issue as packages/ is mainly for mod-managers

[BigSpice]

On it.

[GeckoEidechse]

Aight, so from the looks of it we got a bunch feature PRs queued up that we could release together as v1.17.0. In particular that also includes adding Arc Cannon meaning that this is a breaking update in terms of version compatibility. The way we address version incompatible changes is to up our version gate meaning that everyone is forced to run the new version if they wanna authenticate with master server.

This is also perfect for the change with the packages directory as that means you do not have to consider still supporting installing mods in a legacy form in any way.

 

As we don't wanna delay the Northstar update forever, I'd propose we start rolling out release candidates over the weekend and then release it within the next week.

Now I know that this is a rather short timeframe so I wanna stress that if you cannot support packages directory within that time, any mods installed to mods in the old way is still supported in Northstar. You will just temporarily have reduced compatibility to other mod-managers that might already support the new format.

If you are unable to add support packages any time soon (we're talking like multiple weeks here), please let us know so that we can either try to help out implementing it for you or tell users to (temporarily) switch mod-manager until functionality is implemented.

Essentially I don't want the release to be blocked by mod-manager but also understand that everyone also has a live outside of Northstar and isn't always able to commit time to their mod-manager etc ^^

 

TL;DR: Northstar release with that feature some time next week, release candidate over the weekend, legacy mod install is still supported in Northstar for the time being, if you got any issues with implementing (in time), please reach out <3

[GeckoEidechse]

#503 has been merged and will most likely be released some time next week (most likely towards the weekend).

[GeckoEidechse]

Release candidate is still in the works as it's being held back waiting on a remaining PR. In the meantime, if you need a working build for testing, just download the newest launcher from CI.

[GeckoEidechse]

Release candidate is out by now if anyone needs it for testing and release is slated for Friday.

Just to repeat myself I wanna stress that if you cannot support packages directory within that time, any mods installed to mods in the old way is still supported in Northstar. You will just temporarily have reduced compatibility to other mod-managers that might already support the new format.

[GeckoEidechse]

v1.17.0 has been publicly released so I'm gonna close this issue now.

As mentioned before, the old method still works for mods so you got time to move over. If there are any complications, please reach out <3

[begin-theadventure]

Wouldn't it make more sense to leave the mods in mods and move only the Northstar.X mods to core?

[GeckoEidechse]

Wouldn't it make more sense to leave the mods in mods and move only the Northstar.X mods to core?

That unfortunately won't work given how around 1/4 of all players on Northstar unfortunately still install/update their game manually where the common instruction for updating is to simply overwrite the existing files.

So that would result in duplicated Northstar.XXXXX mods between mods and core.

Now we could add logic to blacklist the Northstar.XXXX mods in mods to remedy this but that still leaves us with the messy situation with mods from Thunderstore where one has to extract subfolders of a mod in Thunderstore and place it in the right location (mods and plugins) while also adding additional information to map the Northstar mod back to it's Thunderstore mod for updating.

And then even more spaghetti code is needed to ensure that updating a mod from Thunderstore properly removes the old one before adding the new one while also making sure not to remove the old one when the new one fails to install.

So yeah, basically packages is the simpler solution but it's also only the first step. Moving core mods out of mods is on our todo list ;)

[GeckoEidechse]

You know, it just now occurred to me that I maybe should've let the Thunderstore people (i.e. @ebkr and @MythicManiac) know of this.

I mean r2mm uses it's own profile so mod-manager compat isn't an issue but it probably even simplifies their own codebase for Northstar if they switch over the support to packages directory.

TL;DR: When downloading mod from Thunderstore, take AUTHOR-MOD-VERSION.zip and stick the contents into <profile folder>/packages/AUTHOR-MOD-VERSION/ as opposed to manually extracting contained mods into <profile folder>/mods previously.