Add script fallback priority; improve language fallback priority

[robertbastian]

#3867

[Manishearth]

seems good to me but would also like shane to take a look

[sffc]

Thanks for working on this! Algorithm needs some tweaks

[robertbastian]

Btw this is mainly inspired by #3972 which has some prior discussion.

[sffc]

• @sffc - There are a few things we should definitly hit:
  • sr-ME
  • sr-Latn-ME (actually no, see below)
  • und-Latn(-ME?)
• @robertbastian I don't think there is ever data in und-Latn-ME. Once we infer the script from the region, we can throw it away.
• @sffc We need to reach sr-Latn-ME even though we don't store data in sr-ME because CLDR sometimes puts data in sr-Latn-ME. We currently run the same fallbacker in both datagen (CLDR land) and runtime (ICU4X land). If we want to change this behavior, the SourceProvider should automatically load sr-Latn-ME data when sr-ME is requested of it.
• @robertbastian That's what it does. See add_script_extended in the Cldr code:

  icu4x/provider/source/src/cldr_cache.rs

  Line 250 in 8be0aec

  if self.0.serde_cache.file_exists(&path)? {

• @sffc OK great. So we then shouldn't need to reach sr-Latn-ME in either language or script fallback, right?
• @robertbastian - We are in language and I'm not in my script PR
• @sffc OK, let's not retain the region then when going to the script
• @sffc Also note that the subdivision is basically part of the region. So we shouldn't retain the subdivision.
• TODO:
  • remove redundant step in language fallback (sr-Latn-ME)
  • remove und-variant step in script fallback (script fallback should end with a script, like region fallback ends with a region)

[sffc]

Observation: script fallback priority ended up being language fallback priority with one or two extra things at the end. We might be able to do some more code sharing in a future cleanup PR.