infra: implement inactive locales content generation enhancement

[xDivisionByZerox]

Description

This PR introduces a locale file normalization into the generateLocales script. This allows us to remove additional maintainace burdens by automatically checking (and fixing) for the following this in the same file:

• duplicated locale entries
• limiting the locale file entries to 1000
• sorting the entries alphabetically

This PR also includes all to changes to all locale files which are affeected by the new rule. Honestly, I have no idea how you want to review this...

How to review

Please follow the steps @ST-DDT suggested in #2265 (comment):

The easiest way to review these is by reviewing the script and then resetting the locales to next and run the script again.
Then there shouldn't be any diff to the committed state.
Finally, some manual tests with some of the files.
So IMO there isn't a need to review them all manually.

[xDivisionByZerox]

The pipeline is failing to locales that were implemented poorly before. Look at:

faker/src/locales/ar/date/weekday.ts

Lines 1 to 13 in 8530a3e

	import type { DateEntryDefinition } from '../../../definitions';
	export default {
	wide: [
	'الأحَد',
	'الإثنين',
	'الثلاثاء',
	'الأربعاء',
	'الخميس',
	'الجمعة',
	'السبت',
	],
	} as DateEntryDefinition;

We simply casted the type here. What should be the expected behaviour here?

[matthewmayer]

I'm not sure blanket sorting everything makes sense

You lose inherent ordering like days of the week.

And in many languages a naive sort based on Unicode character values doesn't reflect how strings are sorted.

Plus it causes a lot of churn meaning git blame is less useful.

[ST-DDT]

IMO we should split these composite files into a folder and individual files.

[xDivisionByZerox]

IMO we should split these composite files into a folder and individual files.

That doesn't really answer the question I had. Should the generation script be able to handle invalid locales or no? Since the DateEntryDefinition requires the keys wide and abbr and we cast it, we currently have invalid data definitions for some locales.

[ST-DDT]

How many invalid entries do we have?
If they are a select few, we should fix them instead of working around them in our code.
Even if we just do them in a non optimal way, it would probably still better than not having them at all. If we cannot find the data we can fall back to [].

[matthewmayer]

I'm not sure blanket sorting everything makes sense

You lose inherent ordering like days of the week.

And in many languages a naive sort based on Unicode character values doesn't reflect how strings are sorted.

Plus it causes a lot of churn meaning git blame is less useful.

Additionally we may be losing useful information in the definitions by resorting.

For example while implementing #1704 it turned out the first_name file actually had all the male names first, then all the female names. So it was easy to split them. If this script had already been run prior to that, there wouldn't have been any easy way to separate the female and male names.

The number of files being touched here makes it unrealistic to manually review each one and see if any useful information is being discarded.

[xDivisionByZerox]

Additionally we may be losing useful information in the definitions by resorting.

That might be. But the thing is that we already request our contributors to sort te locale entries. Some examples:

• feat(locale): add Dutch province abbreviations #2232 (review)
• fix(person): Improvements to Dutch name generation, in particular regarding affixes #1778 (review)
• feat(locale): Add locale ro_MD #1848 (comment)

We generally want this. Sure, we might lose some context but gain standardization in return.

The number of files being touched here makes it unrealistic to manually review each one and see if any useful information is being discarded.

Agreed. But sadly this always happends if we do major changes to the locale structure. What do you suggest?

[ST-DDT]

Additionally we may be losing useful information in the definitions by resorting.

AFAIK there aren't any data in our locales that are supposed to contain any extra information by their order.
There is currently the location.direction data that use the index to distinguish between cardinal and sub-cardinal directions, but IMO that can be easily resolved by splitting the data.

For example while implementing #1704 it turned out the first_name file actually had all the male names first, then all the female names.

Is there a file left that would still support this or all of them already split?
If they are already split, then we don't loose anything. (I'm not aware of any)
If not we can split them before the merge.

The number of files being touched here makes it unrealistic to manually review each one and see if any useful information is being discarded.

The easiest way to review these is by reviewing the script and then resetting the locales to next and run the script again.
Then there shouldn't be any diff to the committed state.
Finally, some manual tests with some of the files.
So IMO there isn't a need to review them all manually.

[xDivisionByZerox]

Also your comment @matthewmayer: #1823 (review)

• The entries should be sorted alphabetical order

I'm not saying that it is not allowed to change ones opinon, but you had this idea once as well^^

[xDivisionByZerox]

The easiest way to review these is by reviewing the script and then resetting the locales to next and run the script again.
Then there shouldn't be any diff to the committed state.

Honestly, thats super big brain. Never thought about this kind of review 👀

[xDivisionByZerox]

How many invalid entries do we have?
If they are a select few, we should fix them instead of working around them in our code.
Even if we just do them in a non optimal way, it would probably still better than not having them at all. If we cannot find the data we can fall back to [].

All that are currently fail in the CI. So 3 AFAICT.

[ST-DDT]

All that are currently fail in the CI. So 3 AFAICT.

IMO 3 are easily fixable manually.
(We should do that in a separate PR though)

[matthewmayer]

I agree new entries should always be in alphabetical order unless they are something with a logical order like days of the week. I'm just concerned about subtly losing information if old entries are sorted alphabetically.

[xDivisionByZerox]

Blocked by #2293

[codecov]

Codecov Report

Merging #2265 (cf5f72e) into next (a193693) will decrease coverage by 0.01%.
The diff coverage is n/a.

Additional details and impacted files

@@            Coverage Diff             @@
##             next    #2265      +/-   ##
==========================================
- Coverage   99.59%   99.58%   -0.01%     
==========================================
  Files        2823     2823              
  Lines      255517   255517              
  Branches     1106     1105       -1     
==========================================
- Hits       254475   254458      -17     
- Misses       1014     1031      +17     
  Partials       28       28              

see 2 files with indirect coverage changes