Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Example of baked data crate with macros #2992

Closed
wants to merge 7 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ members = [
"experimental/casemapping",
"experimental/compactdecimal",
"experimental/displaynames",
"experimental/globaldata",
"experimental/ixdtf",
"experimental/relativetime",
"experimental/tutorials",
Expand Down Expand Up @@ -77,6 +78,7 @@ lto = true
[profile.release-opt-size]
inherits = "release"
opt-level = "s"
debug = true
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(reminder to remove this line)


# Enable debug information specifically for memory profiling.
# https://docs.rs/dhat/0.2.1/dhat/#configuration
Expand Down
14 changes: 14 additions & 0 deletions experimental/globaldata/Cargo.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
[package]
name = "globaldata"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dev-dependencies]
icu_provider = { path = "../../provider/core" }
icu_locid = { path = "../../components/locid" }

[features]
custom_data = []
all_features_hack = []
10 changes: 10 additions & 0 deletions experimental/globaldata/src/data/core_helloworld_v1/bn.data.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
#[macro_export]
macro_rules! data_core_helloworld_v1_bn {
() => {
icu_provider::hello_world::HelloWorldV1 {
message: alloc::borrow::Cow::Borrowed("ওহে বিশ\u{9cd}ব"),
}
};
}

pub use data_core_helloworld_v1_bn;
10 changes: 10 additions & 0 deletions experimental/globaldata/src/data/core_helloworld_v1/en.data.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
#[macro_export]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wrapping each data expression in a macro is a lot of cognitive overhead, and probably also compile time overhead.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The cognitive overhead is being moved from the call site (people need to learn how data providers work) to an internal crate (that only us and power users will see).

macro_rules! data_core_helloworld_v1_en {
() => {
icu_provider::hello_world::HelloWorldV1 {
message: alloc::borrow::Cow::Borrowed("Hello World"),
}
}
}

pub use data_core_helloworld_v1_en;
10 changes: 10 additions & 0 deletions experimental/globaldata/src/data/core_helloworld_v1/ja.data.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
#[macro_export]
macro_rules! data_core_helloworld_v1_ja {
() => {
icu_provider::hello_world::HelloWorldV1 {
message: alloc::borrow::Cow::Borrowed("こんにちは世界"),
}
}
}

pub use data_core_helloworld_v1_ja;
34 changes: 34 additions & 0 deletions experimental/globaldata/src/data/core_helloworld_v1/mod.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
include!("bn.data.rs");
include!("en.data.rs");
include!("ja.data.rs");
include!("ru.data.rs");

#[macro_export]
macro_rules! impl_core_helloworld_v1 {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the plan for AnyProvider? Having one macro per key doesn't work, the keys should be arguments (plural) to the macro.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it should "just work"

struct LocalBakedProvider;
impl_core_helloworld_v1!(LocalBakedProvider);
impl_decimal_symbols_v1!(LocalBakedProvider);
impl_any_provider!(LocalBakedProvider, [
  icu_provider::hello_world::HelloWorldV1Marker,
  icu_decimal::provider::SymbolsV1Marker,
]);

($provider:path) => {
impl DataProvider<icu_provider::hello_world::HelloWorldV1Marker> for $provider {
fn load(&self, req: DataRequest) -> Result<DataResponse<icu_provider::hello_world::HelloWorldV1Marker>, DataError> {
type DataStruct =
<icu_provider::hello_world::HelloWorldV1Marker as icu_provider::DataMarker>::Yokeable;
static KEYS: &[&str] = &["bn", "en", "en-US", "ja", "ru"];
static BN: DataStruct = $crate::baked::core_helloworld_v1::data_core_helloworld_v1_bn!();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This works, but it will be hard to generate because you're using the absolute path of where this will be included, e.g. it has to be included at $crate::baked. If we can't find a way to use relative addressing for the data file (either using mod or include), we should inline the data to step around this problem.

Copy link
Member Author

@sffc sffc Jan 25, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The path is relative to the globaldata crate, which is under our control. We may need to tell databake-datagen what that path is, but this is totally an internal thing.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be useful if the logic we add here is also usable by datagen clients, not only for this crate.

static EN: DataStruct = $crate::baked::core_helloworld_v1::data_core_helloworld_v1_en!();
static JA: DataStruct = $crate::baked::core_helloworld_v1::data_core_helloworld_v1_ja!();
static RU: DataStruct = $crate::baked::core_helloworld_v1::data_core_helloworld_v1_ru!();
static DATA: &[&DataStruct] = &[&BN, &EN, &EN, &JA, &RU];
KEYS.binary_search_by(|k| req.locale.strict_cmp(k.as_bytes()).reverse())
.ok()
.map(|i| unsafe { *DATA.get_unchecked(i) })
.map(zerofrom::ZeroFrom::zero_from)
.map(DataPayload::from_owned)
.map(|payload| DataResponse {
metadata: Default::default(),
payload: Some(payload),
})
.ok_or_else(|| DataErrorKind::MissingLocale.with_req(icu_provider::hello_world::HelloWorldV1Marker::KEY, req))
}
}
};
}

pub use impl_core_helloworld_v1;
10 changes: 10 additions & 0 deletions experimental/globaldata/src/data/core_helloworld_v1/ru.data.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
#[macro_export]
macro_rules! data_core_helloworld_v1_ru {
() => {
icu_provider::hello_world::HelloWorldV1 {
message: alloc::borrow::Cow::Borrowed("Привет, мир"),
}
}
}

pub use data_core_helloworld_v1_ru;
1 change: 1 addition & 0 deletions experimental/globaldata/src/data/mod.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
pub mod core_helloworld_v1;
35 changes: 35 additions & 0 deletions experimental/globaldata/src/lib.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
#![allow(unused_imports)]
#![allow(unused_macros)]

extern crate alloc;

#[doc(hidden)]
pub mod baked {
#[cfg(all(feature = "custom_data", not(feature = "all_features_hack")))]
include!(env!("ICU4X_MACROINCLUDE_PATH"));
#[cfg(any(not(feature = "custom_data"), feature = "all_features_hack"))]
include!("data/mod.rs");
}

#[cfg(test)]
mod tests {
use super::*;
use icu_provider::prelude::*;

#[test]
fn test() {
use baked::core_helloworld_v1::*;
struct BakedProvider;
impl_core_helloworld_v1!(BakedProvider);

assert_eq!(
"こんにちは世界",
icu_provider::hello_world::HelloWorldFormatter::try_new_unstable(
&BakedProvider,
&icu_locid::locale!("ja").into(),
)
.unwrap()
.format_to_string()
);
}
}
1 change: 1 addition & 0 deletions provider/core/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,7 @@ stable_deref_trait = { version = "1.2.0", default-features = false }
log = { version = "0.4", optional = true }
icu_provider_macros = { version = "1.0.0", path = "../macros", optional = true }
serde = { version = "1.0", default-features = false, features = ["derive", "alloc"], optional = true }
globaldata = { path = "../../experimental/globaldata", optional = true }

# Serde formats
serde_json = { version = "1.0", default-features = false, features = ["alloc"], optional = true }
Expand Down
25 changes: 25 additions & 0 deletions provider/core/src/hello_world.rs
Original file line number Diff line number Diff line change
Expand Up @@ -238,6 +238,31 @@ impl HelloWorldFormatter {

crate::gen_any_buffer_constructors!(locale: include, options: skip, error: DataError);

/// Creates a new [`HelloWorldFormatter`] with global data.
///
/// Requires the "globaldata" Cargo feature.
///
/// # Examples
///
/// ```
/// use icu_provider::hello_world::HelloWorldFormatter;
/// use icu_locid::locale;
///
/// let formatter = HelloWorldFormatter::try_new(&locale!("ja").into()).unwrap();
///
/// assert_eq!(
/// "こんにちは世界",
/// formatter.format_to_string()
/// );
/// ```
#[cfg(feature = "globaldata")]
pub fn try_new(locale: &DataLocale) -> Result<Self, DataError> {
struct LocalBakedProvider;
use crate as icu_provider;
globaldata::impl_core_helloworld_v1!(LocalBakedProvider);
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An open question is how to enable vertical fallback. An easy way, which is probably fine, is

let provider = globaldata::with_vertical_fallback!(LocalBakedProvider);

and the macro has two versions: one that is a no-op, and one that is enabled with the globaldata/fallback feature that actually enables the fallback

Self::try_new_unstable(&LocalBakedProvider, locale)
}

/// Formats a hello world message, returning a [`FormattedHelloWorld`].
#[allow(clippy::needless_lifetimes)] // documentary example
pub fn format<'l>(&'l self) -> FormattedHelloWorld<'l> {
Expand Down