Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

I18n #700

Closed
wants to merge 70 commits into from
Closed

I18n #700

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
70 commits
Select commit Hold shift + click to select a range
f8af562
added language files for tests
89Q12 Jun 16, 2022
e9b1be5
added htmls files to test translation
89Q12 Jun 16, 2022
9877915
added tests
89Q12 Jun 16, 2022
2eb0fdf
added parser for localize(foo, bar: baz)
89Q12 Jun 16, 2022
7d11fd2
added tests for the localize parser
89Q12 Jun 16, 2022
8336929
added visit_localize, arm: Expr::Localize, field:localized_messages
89Q12 Jun 16, 2022
72dda5e
added extraction for the locale field and localizer field
89Q12 Jun 16, 2022
858ea69
added locale attribute
89Q12 Jun 16, 2022
465483d
corrected locale attribute
89Q12 Jun 16, 2022
39a1370
added feature localization
89Q12 Jun 16, 2022
47cbb51
added #[cfg(feature = "localization")]
89Q12 Jun 16, 2022
1874729
Fixed errors and added comment
89Q12 Jun 16, 2022
98ff019
fix cargo files
89Q12 Jun 16, 2022
07db0e7
fix test: test_parse_nested_localize
89Q12 Jun 16, 2022
8bd252e
added #![cfg(feature = "localization")] to i18n tests
89Q12 Jun 16, 2022
c96ada6
added askama::Local struct with impl
89Q12 Jun 17, 2022
d1a46fe
changed tests to use askama::Local
89Q12 Jun 17, 2022
cdbc7d0
removed init_translation macro since its not needed anymore
89Q12 Jun 17, 2022
4c7300c
refactored no args test adn its template
89Q12 Jun 17, 2022
7e1f5e6
Merge branch 'main' of https://github.com/11Tuvork28/askama into i18n
89Q12 Jun 17, 2022
17f95e7
removed last todos
89Q12 Jun 17, 2022
2c827f2
Added test for invalid tags with no fallack language
89Q12 Jun 17, 2022
c3f5684
fixed lint error from Lint workflow
89Q12 Jun 17, 2022
2658da3
Fixed typo th -> the
89Q12 Jun 17, 2022
7b89872
fixed test I messed up
89Q12 Jun 17, 2022
5ce38a3
Added fn quoted_ident to support only localize("foo", bar:baz) and
89Q12 Jun 17, 2022
7a11db6
Updated test templates -> quoted all messages
89Q12 Jun 17, 2022
355163a
Added cut() to localze function
89Q12 Jun 17, 2022
62027c8
Revert "Added cut() to localze function" because expr_any uses alt wh…
89Q12 Jun 17, 2022
bbe5991
HashMap<String, ...> to HashMap<&str, ...>
89Q12 Jun 17, 2022
f3ec8c7
removed .to_string()
89Q12 Jun 17, 2022
cc01d9b
removed "dep:" from toml files and removed println
89Q12 Jun 17, 2022
afbc64f
A bunch of changes
Kijewski Jun 17, 2022
b3714f4
Merge pull request #1 from Kijewski/i18n
89Q12 Jun 18, 2022
569a79b
Remove unnecessary feature guard
89Q12 Jun 19, 2022
db5c3e2
Corrected comment
89Q12 Jun 19, 2022
129786d
Validate localization at compile time
Kijewski Jun 19, 2022
ba50d10
Merge pull request #2 from Kijewski/br-i18n-with-compile-time-checks
89Q12 Jun 19, 2022
ca2ca74
Fixed various Clippy complaints
89Q12 Jun 19, 2022
da0319b
added feature guards to make the compiler happy and fix errors
89Q12 Jun 19, 2022
da9f506
changed forbid(unsafe) to forbid(unsafe_code)
89Q12 Jun 19, 2022
96c63c5
Created typ alias PathResources for Vec<(PathBuf, Resource<String>
89Q12 Jun 19, 2022
7663bf8
Fix "all" the clippy warnings
Kijewski Jun 19, 2022
03b4979
Merge pull request #3 from Kijewski/br-i18n-with-compile-time-checks
89Q12 Jun 19, 2022
08d34bd
Merge branch 'djc:main' into i18n
89Q12 Jul 12, 2022
39b0c4a
Merge branch 'main' into i18n
89Q12 Jul 28, 2022
6872a72
Fix lint job
89Q12 Sep 5, 2022
92fadc9
fix: Remove commented out i18n tests
LeoniePhiline Oct 10, 2022
98a4777
fix: Add trailing newline to i18n test templates
LeoniePhiline Oct 10, 2022
63ed5c9
fix: `cargo fmt`, remove trailing whitespace
LeoniePhiline Oct 10, 2022
073e92a
fix: Rename feature `localization` to `i18n`
LeoniePhiline Oct 10, 2022
f944c86
opinionated: Rename initialization macro `localization!` to `i18n_load!`
LeoniePhiline Oct 10, 2022
c32b22d
style: Change wording "have to" to "need to"
LeoniePhiline Oct 10, 2022
426575e
refactor: Confine i18n code inside i18n modules
LeoniePhiline Oct 10, 2022
81cce1a
fix: Hide dependency `fluent_templates` from library users
LeoniePhiline Oct 10, 2022
1c315d6
fix(deps): Update dependency `fluent-templates` to `0.8.0`
LeoniePhiline Oct 10, 2022
6384ae2
docs: Add initial i18n module documentation
LeoniePhiline Oct 10, 2022
7f82494
Merge pull request #4 from LeoniePhiline/i18n
89Q12 Oct 11, 2022
3f002ef
docs: Rename example template struct
LeoniePhiline Oct 11, 2022
925d1b5
Merge branch 'djc:main' into i18n
89Q12 Oct 11, 2022
8f30ba5
Merge pull request #5 from LeoniePhiline/i18n
89Q12 Oct 11, 2022
b52aaf3
fix: Implement missing recursive `is_cachable()` for `Expr::Localize`
LeoniePhiline Oct 11, 2022
9909c36
fix(style): Name localization message identifier as in Fluent Project
LeoniePhiline Oct 11, 2022
6cda9f5
fix(style): Normalize test names, prefix all with `test_`
LeoniePhiline Oct 11, 2022
9e1f326
Merge pull request #6 from LeoniePhiline/i18n
89Q12 Oct 11, 2022
7c15904
fix: Add missing trailing newline
LeoniePhiline Oct 11, 2022
5c35e12
Merge branch '11Tuvork28:i18n' into i18n
LeoniePhiline Oct 11, 2022
2bbba70
fix: Compilation without feature `i18n` failed
LeoniePhiline Oct 11, 2022
b5dda4c
Merge pull request #7 from LeoniePhiline/i18n
89Q12 Oct 11, 2022
f9dfa60
Merge branch 'djc:main' into i18n
89Q12 Jan 12, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions askama/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ maintenance = { status = "actively-developed" }
default = ["config", "humansize", "num-traits", "urlencode"]
config = ["askama_derive/config"]
humansize = ["askama_derive/humansize", "dep_humansize"]
i18n = ["askama_derive/i18n", "fluent-templates", "parking_lot"]
markdown = ["askama_derive/markdown", "comrak"]
num-traits = ["askama_derive/num-traits", "dep_num_traits"]
serde-json = ["askama_derive/serde-json", "askama_escape/json", "serde", "serde_json"]
Expand All @@ -43,6 +44,8 @@ askama_escape = { version = "0.10.3", path = "../askama_escape" }
comrak = { version = "0.15", optional = true, default-features = false }
dep_humansize = { package = "humansize", version = "2", optional = true }
dep_num_traits = { package = "num-traits", version = "0.2.6", optional = true }
fluent-templates = { version = "0.8.0", optional = true }
parking_lot = { version = "0.12.1", optional = true }
percent-encoding = { version = "2.1.0", optional = true }
serde = { version = "1.0", optional = true, features = ["derive"] }
serde_json = { version = "1.0", optional = true }
Expand Down
129 changes: 129 additions & 0 deletions askama/src/i18n.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,129 @@
//! Module for compile time checked localization
//!
//! # Example:
//!
//! [Fluent Translation List](https://projectfluent.org/) resource file `i18n/es-MX/basic.ftl`:
//!
//! ```ftl
//! greeting = ¡Hola, { $name }!
//! ```
//!
//! Askama HTML template `templates/example.html`:
//!
//! ```html
//! <h1>{{ localize("greeting", name: name) }}</h1>
//! ```
//!
//! Rust usage:
//! ```ignore
//! use askama::i18n::{langid, Locale};
//! use askama::Template;
//!
//! askama::i18n::load!(LOCALES);
//!
//! #[derive(Template)]
//! #[template(path = "example.html")]
//! struct ExampleTemplate<'a> {
//! #[locale]
//! loc: Locale<'a>,
//! name: &'a str,
//! }
//!
//! let template = ExampleTemplate {
//! loc: Locale::new(langid!("es-MX"), &LOCALES),
//! name: "Hilda",
//! };
//!
//! // "<h1>¡Hola, Hilda!</h1>"
//! template.render().unwrap();
//! ```

use std::collections::HashMap;
use std::iter::FromIterator;

// Re-export conventiently as `askama::i18n::load!()`.
// Proc-macro crates can only export macros from their root namespace.
/// Load locales at compile time. See example above for usage.
pub use askama_derive::i18n_load as load;

pub use fluent_templates::{self, fluent_bundle::FluentValue, fs::langid, LanguageIdentifier};
use fluent_templates::{Loader, StaticLoader};
use parking_lot::const_mutex;

pub struct Locale<'a> {
loader: &'a StaticLoader,
language: LanguageIdentifier,
}

impl Locale<'_> {
pub fn new(language: LanguageIdentifier, loader: &'static StaticLoader) -> Self {
Self { loader, language }
}

pub fn translate<'a>(
&self,
msg_id: &str,
args: impl IntoIterator<Item = (&'a str, FluentValue<'a>)>,
) -> Option<String> {
let args = HashMap::<&str, FluentValue<'_>>::from_iter(args);
let args = match args.is_empty() {
true => None,
false => Some(&args),
};
self.loader.lookup_complete(&self.language, msg_id, args)
}
}

/// Similar to OnceCell, but it has an additional take() function, which can only be used once,
/// and only if the instance was never dereferenced.
///
/// The struct is only meant to be used by the [`i18n_load!()`] macro.
/// Concurrent access will deliberately panic.
///
/// Rationale: StaticLoader cannot be cloned.
#[doc(hidden)]
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems like a bunch of dark magic. Why do we need this?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Kijewski is the author of this section. I am sure they can tell you the details.

All I can tell is that this is an adjustment / workaround to an implementation of fluent_templates::static_loader!, which is used to include the localization data into the compiled binary, and to validate all calls to the localize() template expr at compile time.

pub struct Unlazy<T>(parking_lot::Mutex<UnlazyEnum<T>>);

enum UnlazyEnum<T> {
Generator(Option<fn() -> T>),
Value(Box<T>),
}

impl<T> Unlazy<T> {
pub const fn new(f: fn() -> T) -> Self {
Self(const_mutex(UnlazyEnum::Generator(Some(f))))
}

pub fn take(&self) -> T {
let f = match &mut *self.0.try_lock().unwrap() {
UnlazyEnum::Generator(f) => f.take(),
_ => None,
};
f.unwrap()()
}
}

impl<T> std::ops::Deref for Unlazy<T>
where
Self: 'static,
{
type Target = T;

fn deref(&self) -> &Self::Target {
let data = &mut *self.0.try_lock().unwrap();
let value: &T = match data {
UnlazyEnum::Generator(f) => {
*data = UnlazyEnum::Value(Box::new(f.take().unwrap()()));
match data {
UnlazyEnum::Value(value) => value,
_ => unreachable!(),
}
}
UnlazyEnum::Value(value) => value,
};

// SAFETY: This transmutation is safe because once a value is assigned,
// it won't be unassigned again, and Self has static lifetime.
unsafe { std::mem::transmute(value) }
}
}
4 changes: 3 additions & 1 deletion askama/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -59,13 +59,15 @@
//! in the configuration file. The default syntax , "default", is the one
//! provided by Askama.

#![forbid(unsafe_code)]
#![cfg_attr(not(feature = "i18n"), forbid(unsafe_code))]
#![deny(elided_lifetimes_in_paths)]
#![deny(unreachable_pub)]

mod error;
pub mod filters;
pub mod helpers;
#[cfg(feature = "i18n")]
pub mod i18n;

use std::fmt;

Expand Down
7 changes: 5 additions & 2 deletions askama_derive/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,12 @@ proc-macro = true
[features]
config = ["serde", "toml"]
humansize = []
i18n = ["fluent-syntax", "fluent-templates", "serde", "toml"]
markdown = []
urlencode = []
num-traits = []
serde-json = []
serde-yaml = []
num-traits = []
urlencode = []
with-actix-web = []
with-axum = []
with-gotham = []
Expand All @@ -30,6 +31,8 @@ with-tide = []
with-warp = []

[dependencies]
fluent-syntax = { version = "0.11.0", optional = true, default-features = false }
fluent-templates = { version = "0.8.0", optional = true, default-features = false }
mime = "0.3"
mime_guess = "2"
nom = "7"
Expand Down
33 changes: 33 additions & 0 deletions askama_derive/src/generator.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1291,9 +1291,42 @@ impl<'a> Generator<'a> {
Expr::RustMacro(name, args) => self.visit_rust_macro(buf, name, args),
Expr::Try(ref expr) => self.visit_try(buf, expr.as_ref())?,
Expr::Tuple(ref exprs) => self.visit_tuple(buf, exprs)?,
#[cfg(feature = "i18n")]
Expr::Localize(ref msg_id, ref args) => self.visit_localize(buf, msg_id, args)?,
})
}

#[cfg(feature = "i18n")]
fn visit_localize(
&mut self,
buf: &mut Buffer,
msg_id: &Expr<'_>,
args: &[(&str, Expr<'_>)],
) -> Result<DisplayWrap, CompileError> {
let localizer =
self.input.localizer.as_deref().ok_or(
"You need to annotate a field with #[locale] to use the localize() function.",
)?;

buf.write(&format!(
"self.{}.translate(",
normalize_identifier(localizer)
));
self.visit_expr(buf, msg_id)?;
buf.writeln(", [")?;
buf.indent();
for (k, v) in args {
buf.write(&format!("({:?}, ::askama::i18n::FluentValue::from(", k));
self.visit_expr(buf, v)?;
buf.writeln(")),")?;
}
buf.dedent()?;
// Safe to unwrap, as `msg_id` is checked at compile time.
buf.write("]).unwrap()");

Ok(DisplayWrap::Unwrapped)
}

fn visit_try(
&mut self,
buf: &mut Buffer,
Expand Down
Loading