Extract locales from downloads

[HebaruSan]

Motivation

#2113 suggested filtering mods by language. To achieve that, we would need to acquire metadata tracking modules' translations. That would be pretty tedious to do manually.

Changes

Now Netkan will examine the .cfg files in each download to see if they have translation data using some fancy regular expressions. If any translations are found, their locale names will be added to the module's localizations property. This is a list of strings that flows through to the CkanModules.localizations property in the application and could be used to implement displays and filters in the future. No UI changes are made yet.

I'm mainly seeking feedback on this at this point. Did I miss anything that would prevent this from working? Would we need additional data to implement filters?

[DasSkelett]

Feedback-wise I can only attest that this works with every mod I tested so far, and that it is a cool feature :D.

For a later GUI implementation, I think we have to redo our search/filter structure soon.
Either like you suggested in #2736 and include the locale filter in the dropdown,
or create a completely new dialog for advanced searches, and move everything but a simple name/identifier search box there.
But I think that discussion is better held in #2736.

[HebaruSan]

It's probably better to call the property localizations or translations instead of locales.

• Localizations is the keyword for the parent node in the .cfg files:

  Localization
  {
          en-us
          {
  

• locale doesn't really explain what is provided by the module, whereas translations implies that the module is translated into those locales and localizations implies that it is localized into them
• locale kind of implies that the values should conform to ISO/IEC_15897 (which KSP does not do—a dash replaces the underscore, and the country codes are lowercase and sometimes omitted):
  https://en.wikipedia.org/wiki/Locale_(computer_software)

  On POSIX platforms such as Unix, Linux and others, locale identifiers are defined by ISO/IEC 15897

Comparing those two options, localizations is more generic than translations; a translation implies rewriting words in the new language, whereas localizations would include other things like date formats, images, right-to-left, etc.

[cake-pie]

Now Netkan will examine the .cfg files in each download to see if they have translation data using some fancy regular expressions. ... I'm mainly seeking feedback on this at this point. Did I miss anything that would prevent this from working?

1. What if a mod has packaged its localization files as optional downloads in separate file(s)? Will netkan bot look for them automatically (on github, spacedock, etc.) or will additional metadata be required in *.netkans to tell it where to look (and download)?

2. Will this support cases where a localization patch requires replacing certain files from the main download? Essentially this would involve separate downloads as per (1) above -- you'd install the main package first, and then run a localization patch that might overwrite some of the original files.

   Case in point, squad's localization patches do replace KSPedia files as well as some part textures that have text on them. No reason why a mod couldn't do the same.

3. While it is pretty neat to automatically scan for localization entries in *.cfg files, here are a couple of issues to consider:

   • some mods may have placeholder localization files which may be marked as a certain language, but actually just contain English strings, pending actual translation work. It would be misleading to advertise them as having support for the language.
   • considering that textures, kspedia, etc may also be localized, there could conceivably be mods that have such localized files, but no localization entries in *.cfg files. Such mods would not get picked up by the automated process, but it looks like they would still be supported, albeit the metadata would need to be explicitly manually added -- can you confirm?

[HebaruSan]

What if a mod has packaged its localization files as optional downloads in separate file(s)? Will netkan bot look for them automatically (on github, spacedock, etc.) or will additional metadata be required in *.netkans to tell it where to look (and download)?

I imagine we would set up a separate module for each of those optional downloads and mark them as recommended or suggested. Has anyone taken that approach yet? Seems like it would be a big packaging hassle.

Will this support cases where a localization patch requires replacing certain files from the main download?

I think this would have to be handled as per below...

albeit the metadata would need to be explicitly manually added -- can you confirm?

Yes, you could put a localizations property in a netkan to handle a module with localizations that can't be detected automatically. Which reminds me, such values should not be overwritten by the automated extractor, new commit pushed to account for this. Thanks for your thoughts!

[cake-pie]

I imagine we would set up a separate module for each of those optional downloads and mark them as recommended or suggested. Has anyone taken that approach yet? Seems like it would be a big packaging hassle.

AirlockPlus has had localization cfgs for other languages in a separate zip file ever since localization became a thing in KSP -- it was one of the first to have support for all four languages at the time.

The packaging is not that big of a hassle, really.
In my case it's partly because I'm kind of a stickler about not having to download/install optional stuff, support patches, etc. unless the end-user actually needs / wants them. Though it does yield a nice benefit in that I can get stats about how many people are downloading support for other languages.

OTOH the use case where localization involves replacing textures, kspedia, etc, would probably necessitate separate downloads. As far as I can imagine it'd be most aptly handled using the main install + patch approach. Instructions for manual installs would be straightforward: "drop main install into GameData, then drop desired language patch into GameData, accept if it asks about overwriting files". CKAN installs can be configured as needed (looks like a good fit for PROVIDES).

The alternatives I can think of are:

1. Distribute one download that contains all the language options.

   • Manual installation requires more complex instructions that detail how to pick out and install the files for the language you want.
   • Can CKAN handle different install directives depending on desired language?
   • Potential download file bloat if many textures, etc.

2. Have completely separate installation files for each language.

   • Actually really straightforward for manual installs
   • CKAN would still have to treat each language as a separate module.
   • Excessive if only a small fraction of textures etc need replaced.

If language patches offered as separate installs are treated as separate modules, then it pretty much boils back down to one of the use cases I posted at #538 (comment) for auto-suggesting optional localization patches based on the end-user's installed KSP language.

[HebaruSan]

AirlockPlus has had localization cfgs for other languages in a separate zip file

Heh, that file is less than 3 KB. We could already create an AirlockPlus-Localization module and make it a suggestion for AirlockPlus (mental note to self...), and then this pull request's code would mark it as containing the appropriate localizations. So that use case is covered.

Manual installation requires more complex instructions that detail how to pick out and install the files for the language you want

• Can CKAN handle different install directives depending on desired language?

This case would probably be handled by one module per locale, sharing downloads but with different install stanzas. And since this pull request inspects the files to be installed for the given module rather than all the files in the ZIP, each such module would be tagged with the appropriate locales automatically if it contains .cfg files, or manually in the netkan if it's just other resources. So I think this is covered as well.

Have completely separate installation files for each language.

• CKAN would still have to treat each language as a separate module.

Yeah, that's no problem. 👍

[HebaruSan]

This is ready to merge, barring discovery of functional issues or discussion of the schema/spec changes.