Analysis of bevy_fluent for upstreaming

[aecsocket]

Internationalization (i18n) is important but often overlooked in games, so it's important that Bevy does this right. A 3rd party crate for i18n already exists, bevy_fluent, and this issue describes my thoughts on the crate.

This writeup is done on version 0.8.0 of the crate.

Getting started

First thing to point out is the lack of documentation:

• descriptions of structs and modules are very brief
• readme lacks much depth
• no examples in the docs (there are some in examples/ though)

There's very little documentation right now on how to actually use the i18n features, so the best place to start is the examples. A slimmed-down version of the minimal example looks like:

use bevy_fluent::{Locale, FluentPlugin};
use unic_langid::langid; // !!!
use fluent_content::Content; // !!!
// use ...

fn main() {
    App::new()
        // ...
        .add_plugins(FluentPlugin)
        .insert_resource(Locale::new(langid!("en-US")))
        .add_systems(Startup, localized_hello_world)
        .run();
}

fn localized_hello_world(
    localization_builder: LocalizationBuilder,
    asset_server: Res<AssetServer>,
    mut handle: Local<Option<Handle<LoadedFolder>>>,
    mut localization: Local<Option<Localization>>,
) {
    let handle = &*handle.get_or_insert_with(|| asset_server.load_folder("locales"));
    if let Some(LoadState::Loaded) = asset_server.get_load_state(handle) {
        let localization = localization.get_or_insert_with(|| localization_builder.build(handle));

        info!("{}", localization.content("hello-world"));
    }
}

I experienced some friction with this initial, relatively simple example:

• With bevy_fluent alone, this example will not compile. I also needed to add unic_langid and fluent_content as dependencies to the crate.
  • fluent_content should probably be re-exported; unic_langid could be as well but I don't know. If it's in the public API then it should be, right?
• This might just be an issue with Bevy's assets ergonomics, but the localized_hello_world system also does a bunch of stuff relating to loading the i18n assets. This adds a ton of noise to the code for what should be a relatively simple example.
  • The Localization derives Resource, so an example showing how you can just add Res<Localization> would be useful here
• There seems to be no way to add arguments to the content call, which is however supported in fluent::FluentBundle as well as the fluent::fluent_args! macro. Imo it's very important to have an easy-to-use and frictionless formatting mechanism with arguments here. However I like the relative simplicity of localization.content("message-key") here.

In general, I like the approach of having a single Locale resource, and using a Localization to store the actual translation content, since this feels like a standard approach that an app would use:

fn localized_hello_world(i18n: Res<Localization>) {
    info!("{:?}", i18n.content("hello-world"));
}

Architecture

Quickly reading through the source I like the general architecture, it's fairly simple and the module names are self-descriptive (but again, more docs would be very useful).

I'm not a fan of some of the naming like BundleAsset, as this is vague in meaning (the name doesn't relate to i18n in any way) and conflicts with Bevy's existing Bundle terminology (i.e. a bundle of components). In a fully qualified name like bevy_fluent::BundleAsset it makes more sense, but if this were to be upstreamed, it should probably be renamed to make its purpose as an i18n tool clear.

The "fallback chain" mechanism is somewhat unclear - it's not immediately obvious where this is configured, and how it works exactly.

The crate seems to be coupled tightly to folders - I honestly don't know if this is a good idea or not. It limits flexibility, but then again most people will be loading translations from folders.

Overall, I think the crate is in a pretty good state. My priority for this would be:

• docs
• clean up the examples and add comments to them to make it explicitly clear what's going on, and why things are done this way (i.e. what does the Locale resource do?)
• making the process of integrating i18n as frictionless as possible for devs - if there's a lot of annoyances involved in translating strings, then devs simply won't do it
  • this includes making setup of i18n super simple, using i18n in systems, and passing arguments into messages

[alice-i-cecile]

[4:11 PM]TIMMÄH: I think there are multiple separate parts to this:
Locale management, i.e. to select the current locale and handle fallbacks (easy)
General asset localization, i.e., being able to provide localized components for any asset file. We should be able to localize images/textures, videos, audio, etc)
Text localization. This should likely use the foundations built by 2 and also provide Fluent integration, e.g. in the form of an asset loader and integrating licely with bevy_ui
[4:11 PM]Boris: 2 sounds like the hardest to implement right now
[4:11 PM]Boris: I can't think of a good way that you'd localize entire textures or videos in the engine
[4:14 PM]TIMMÄH: For 2, I really want to experiment with setting up an asset source which can handle localized assets, so that you can just write something like asset_server.load("localized://your_image.png") and it will load your_image/en.png if english is selected as active locale.
Unfortunately I don't have so much time right now to work on this and ran into some lifetime issues in an experiment.
But I think this would give us the best UX:
You still get hot reloading and all that stuff even when using localization and to localize your game all you have to do is to replace your asset by a folder with the localized versions and add localized:// to your load call

@TimJentzsch on Discord

[TimJentzsch]

My take on this is that it would be great to have built-in localization support, but that the current approaches (bevy_fluent, bevy_mod_localizaton, etc.) are not ergonomic enough yet.

I think the best thing to integrate into Bevy at this stage would be the basic ingredients locale management. For example, a Locale resource, which stores the currently selected locale and some fallback mechanisms.
This would allow plugins like bevy_fluent, bevy_mod_localization, yarn-slinger, etc. to build on the same foundations (and be interoperable), while not committing to a particular implementation yet.

Then we can experiment on making localization more ergonomic and better integrated into Bevy's ECS.
For example, by leaning more heavily on components or experimenting with asset sources (which I am trying for bevy_mod_localization right now).

[aecsocket]

This sounds like the right approach to me: build a solid base which allows interop between different i18n crates, while not committing to a final implementation yet. Probably would use unic_langid for this Locale resource.

Having a single global Locale sounds reasonable to me for 99% of use cases, and I can't think of one which would require two "primary" locales - maybe separate voiceover/subtitles/UI locales?

[wilk10]

I have just integrated bevy_fluent into my game and I wanted to share my experience as a user.

Overall I agree with the analysis made above. I think it’s a very good starting point to help Bevy support localization as a first party solution. I agree that docs are very much needed and some struct naming could be improved, but overall I was able to integrate it in about a day's worth of work and going forward it’ll be much easier for me to extend the currently supported languages and add new ones. The examples especially hold it back a little, with more focussed ones it would be much clearer in my opinion how it works.

One correction that I would like to do is that it does support passing arguments when retrieving a message. It does that via the fluent_content crate from the same author. I have not tested selectors yet, but will do so soon and report back if I have issues. However, I needed to build my own logic to combine attributes and arguments into the necessary request that fluent_content requires. I very much prefer the approach used by bevy_mod_localization here with try_get_message, try_format_message. I haven’t used it directly though, I just read the code and it seems cleaner.

Another thing that I find not ideal with bevy_fluent is that in order to actually change the language, it requires rebuilding and reinserting the Localization resource via the LocalizationBuilder SystemParam (see here). Again, I very much prefer the approach taken by bevy_mod_localization here with Locale::set() instead.

Overall though, I think bevy_fluent is a very good starting point and with some help from some more user-friendly APIs from bevy_mod_localization I think upstreaming into Bevy will provide a lot of value for users.

[Exanc]

Providing some common basis for interoperability between i18n systems seems like a great idea !

After some tinkering I have come up with an idea that allows the crates to embrace the ECS while providing a common basis on how to handle i18n, starting with the consensus on the usage of unic_langid I defined a resource to hold the current chosen locality for the game:

#[derive(Resource)]
pub struct Locale {
    current: unic_langid::LanguageIdentifier,
    fallback: unic_langid::LanguageIdentifier,
}

Having some experience with integrating i18n in applications the first thing I noticed was how verbose the current solutions are, the current consensus for popular systems is to provide one simple function (ex: $(key, argv*), _(key, argv*) or i18n(key, argv*)) that either returns a localized resource or it's key as a last resort.

Trying to define a simple use case, I came up with this example:

fn hello_world_system (
    localization: Res<TextLocalization>
) {
    let i18n = localization.use_localization();

    println!( "In your language we say hello like: {}", i18n("hello") );
}

You might notice, the TextLocalization, this is because here we don't limit the crate on what type of asset they want to localize while providing a base trait for seamless usage:

pub trait Localization<T, K>
{
    fn use_localization (&self) -> impl Fn(K) -> T;
}

impl Localization<String, &'static str> for TextLocalization {
    fn use_localization (&self) -> impl Fn(&'static str) -> String {
        return |key: &'static str| {
            self.translate(key)
        }
    }
}

This trait not only allows freedom of type but also error handling. While providing pretty much no verbose for the end-user.

In brief, this could provide a shy but strong and flexible basis for i18n within bevy.

As for the loading of resources, bevy_mod_localization's solution seems adequate apart from the declaration of LocalisationFolders which seems superficial to me as we could define as base folder like assets/locales/${lang_id} which is common for applications.

Since, the Locale resource is also defined within bevy we also need to define a way to notice systems of a change, like the queuing the happens when invoking NextSate::set() for systems to preemptively load resources while the transition is happening.

[Exanc]

I do wonder though how it would play with change detection and hot reloading.

I have the same concerns, especially after reading through Bevy UI and Scene Evolution Proposal. Reactiveness seems to be the greatest problem right now.

[Exanc]

I think it could make sense to wrap the unic_langid type in a custom wrapper, which makes the internals opaque, but adding From implementations for convenience.

It's true that making the internals of unic_langid opaque could be convenient as we might not want to support every single feature it provides and exposing extra-Bevy types is not good practice.
But with this solution, my main focus is keeping everything flexible for the crates actually doing the job.

I'm going to iterate more on my idea and see were it goes!

[wilk10]

Regarding the i18n function: I think the concept is interesting, especially because of the familiarity from other contexts, as you have mentioned.
I do wonder though how it would play with change detection and hot reloading.
For example, bevy_ui currently uses retained mode, so nodes are created once and not necessarily updated every frame. So it might be tricky (but not unsolvable) to make sure that all text updates when the locale changes.

Regarding the Localization trait, I ended up implementing something similar in my game, where anything (when reasonable) can implement it and show up correctly localised.

I see the concern, but changing the locale is usually done in a completely separate settings menu. On "Apply" only that menu would need to be rebuilt (and in many apps / games the locale changes only when you actually leave that menu). Afterwards, while navigating back to the main screen / play mode, the UI will respawn correctly localised following the usual flow of the game / app, nothing will need to be rebuilt just for the locale change. I reckon the case where a menu that lets you change the locale is on screen at the same time as a lot of UI elements to be very much an exception imo.

[aecsocket]

I can think of an example where this does happen. In Disco Elysium there's an option to set a keybind to toggle your current text language on the fly. You set a "language 1" and "language 2" in the settings, and use the keybind to toggle between them instantly. I think this is a perfectly valid use case, and should be possible with Bevy's built in localisation.

[Exanc]

I think this is a perfectly valid use case, and should be possible with Bevy's built in localisation.

Yeah, I don't see this as an edgecase but more as a convenience feature. I can see myself using it to validate translations ingame.

[Exanc]

branch: https://github.com/Exanc/bevy/tree/prototype/interoperable-i18n
commit: Exanc@f76820a

I18n interoperability

Just commited to my fork of bevy a first iteration of what I think could be a basis of Bevy native I18n.

This first cornerstone brings a few things:

• Standard way of creating and reading/writing locales inside of Bevy, using the new Locale<Languages> ressource.
  • The new Languages generic type is here to define a set of languages for the application, either statically via enums or dynamicaly via structs. It also allows for multiple Locales with different language sets.
  • Locales both include an main: Option<Language> and a mandatory fallback: Language.
  • The Language struct wraps around unic_langid::LanguageIdentifier.
• Standard way of accessing localized assets with the new Locale::to_locale_asset_path, prepending the Language's name to the path.

asset_server.load(locale.to_locale_asset_path("mani_menu.txt", false)).

• A new Localization trait defining how ressources should be localized by the end-user.

pub trait Localization<R, Languages>
where
    Languages: Default,
    Language: From<Languages>,
    Self: Resource,

The aimsof this first cornerstone is to standardise how we want the end-user (ie: game developers) to localize their assets wihle leaving freedom of implementation for crate developers.

An example is also available: https://github.com/Exanc/bevy/blob/prototype/interoperable-i18n/examples/i18n/localization.rs

Next step

I would like to get some feedback from you before commiting anymore to it. If you have any idea how we could expand it, or if something doesn't seem right.

Mentions: @wilk10 @aecsocket @TimJentzsch

[Exanc]

There should be a default language for the game / app, but then the fallback language should be set based on which language it’s being requested (eg: if fr-CA is being requested, but only fr-FR is available, fallback to fr-FR, not to en-US, if that’s the default language).

The fallback language is left to the end-user's discretion but yes my example is not the best 😂Maybe we should transform Locale into a trait, to leave some more freedom ?

Fluent handles all of that, so Bevy should just support this.

Currently the objective is to give a common basis while keeping the freedom of implementation, that why I chose not to directly use Fluent features (if I understand your point correctly). The fallback language can be changed at any time but you are making a good point, we should let implementors choose how to handle fallback/chains of fallbacks.

In Fluent's syntax there’s messages, variables (+ their value), and attributes that are used to retrieve / resolve a localisation, not only key and arguments, though if Bevy's support is more general, maybe what is implemented here is enough? Maybe it's just a matter of having some more complex example.

You're right! My example is really simple, that's why I left <Self as Localization>::Argv as an unconstrained type, some solutions might only required a key/value pair, but it can be anything from an empty tuple to a full abstract syntax tree if you so desire.

In my implementation, the Localization trait can be implemented for any type. So it’s more granular...

I see were you're going, I chose to constrain implementations of Localization to ressources as a choice of user experience. Could you please detail what was your choice of implementation ?

[Exanc]

For what it's worth, I will work on the second iteration next week. This time taking into consideration the current solutions available for the bevy engine and see if this proposal can be expanded further without constraining implementations.

Thank you all for the feedback.

[wilk10]

Thanks for the reply!

Currently the objective is to give a common basis while keeping the freedom of implementation, that why I chose not to directly use Fluent features

Yes definitely! I think the main point is that the fallback should be per requested locale (however that is set), while the default / main locale should be per app / game. Maybe the fallback doesn't even need to be cached, it's just calculated when a new a new locale is requested, and then the result of the calculation becomes the current locale.

Could you please detail what was your choice of implementation ?

In my game I have something like this:

pub enum WorldResource {
    Coal,
    Food,
    IronOre,
    Steel,
    Stone,
    Wood,
}

impl Localise for WorldResource {
    fn message(&self) -> &'static str {
        match self {
            Self::Coal => "coal",
            Self::Food => "food",
            Self::IronOre => "iron-ore",
            Self::Steel => "steel",
            Self::Stone => "stone",
            Self::Wood => "wood",
        }
    }
}

where the &'static str matches the Fluent message used to retrieve the translation in the FTL file:

coal = Coal
food = Food
iron-ore = Iron Ore
steel = Steel
stone = Stone
wood = Wood

and Localise looks more or less like (where fluent::Localization comes from bevy_fluent):

pub trait Localise {
    fn message(&self) -> &'static str;

    fn attribute(&self) -> Option<&'static str> {
        None
    }

    fn argument_keys(&self) -> Vec<&'static str> {
        Vec::new()
    }

    fn key_values(&self, values: Vec<String>) -> String {
        debug_assert_eq!(values.len(), self.argument_keys().len());
        let keys = self.argument_keys();
        let mapped = keys
            .iter()
            .zip(values.iter())
            .map(|(key, value)| format!("{key}={value}"));
        itertools::Itertools::intersperse(mapped, "&".to_string()).collect::<String>()
    }

    fn format_request(&self, arg_vals: Option<Vec<String>>) -> String {
        let message = self.message();
        match (self.attribute(), arg_vals) {
            (Some(attribute), Some(values)) => {
                let key_values = self.key_values(values);
                format!("{message}.{attribute}?{key_values}")
            }
            (None, Some(values)) => {
                let key_values = self.key_values(values);
                format!("{message}?{key_values}")
            }
            (Some(attribute), None) => format!("{message}.{attribute}"),
            (None, None) => message.to_string(),
        }
    }

    fn localise(&self, i18n: &fluent::Localization, arg_vals: Option<Vec<String>>) -> String {
        let request = self.format_request(arg_vals);
        i18n.content(&request).unwrap()
    }
}

It has the advantage of being very granular and works for every type. Maybe a centralised resource is better for keeping everything tight together though. But the above trait should be more of a responsibility of bevy_fluent or its equivalent, and Bevy should just offer the abstraction to connect everything together.

It does suffer from the concerns raised previously though that it doesn't work well with reactivity and requires the UI to be despawned and respawned when the app/game locale is changed.

[TimJentzsch]

I don’t fully understand the difference between main and fallback language. Fluent offers their own NegotiationStrategy and a negotiate_languages function. There should be a default language for the game / app, but then the fallback language should be set based on which language it’s being requested (eg: if fr-CA is being requested, but only fr-FR is available, fallback to fr-FR, not to en-US, if that’s the default language). Fluent handles all of that, so Bevy should just support this.

I think one thing we need to be mindful of that we need to support the localization of more than just text in the future, e.g. also videos for cut-scenes, image textures, etc.
The fallback mechanism that fluent implements sounds nice, but we should ensure that we can use it in other contexts as well

[Exanc]

@wilk10

Maybe the fallback doesn't even need to be cached, it's just calculated when a new a new locale is requested, and then the result of the calculation becomes the current locale.

You're right, the current solution is quite limited, I quite like the idea of a fallback-chain proposed earlier. We could change the Locale ressource to be more like:

pub struct Locale<Languages> {
  phantom: PhantomData<Languages>,
  current: Language,
  fallback_mechanism: Box(dyn FallbackChain)
}

Where FallbackChain is a trait that defines how a fallback could be determined or not. Leaving the choice of either having a full fallback chain or not (some poeple might preffer to just have a warning).

In my game I have something like this...

Our sollutions are very similar, your WorldResource could be placed in <Self as Localization>::Argv and you will obtain the same machanism. The Localization trait is a Ressource because its supposed to hold a reference to it's translations. I think you just did the same thing but with extra steps.
Your solution might make it harder to have multiple type of localizable asset and make it more verbose for the end-user.