API for accessing Unicode properties

[sffc]

Sub-issues:

• CodePointTrie - polish and move to utils #1070
• Define property enums/newtypes, including AsULE impls #1071
• CodePointTrie data provider struct definition #1072
• CodePointTrie transformer code (data source to/from provider struct) #1073
• CodePointTrie API function plumbing #1074
• Plug uprops into icu4x-datagen and add examples using icu_testdata::get_provider() #936

---

With UnicodeSet (#91) and UCPTrie (#132) coming along, we should start thinking about what the API will look like for accessing Unicode properties.

A simple and clean solution would be a bunch of functions returning either UnicodeSet or UCPTrie, such as:

// Binary Unicode property
pub fn get_whitespace_set() -> Result<UnicodeSet, Error>;

// Enumerated Unicode property
pub fn get_digits_trie() -> Result<UCPTrie, Error>;

These functions would pull from the data provider. The data provider produces serialized sets or tries, and these functions are pretty thin wrappers that convert the serialized format to a Rust UnicodeSet or UCPTrie.

Thoughts?

@markusicu @macchiati @srl295 @EvanJP

[sffc]

@kpozin suggested that we could also have an API on char, similar to toLocaleString(). It might be more convenient when getting a property for a single character.

@zbraniecki suggested looking at prior art like "is_ascii_whitespace" in the standard library.

[aethanyc]

@echeran I'm looking for the API to query the line break property value given a codepoint, e.g. f(codepoint, line_break_property) -> line_break_property_value, and investigating if it's possible to replaced the API currently used in line breaker.

icu4x/experimental/segmenter/src/line_breaker.rs

Lines 72 to 76 in cd4b7c5

	#[inline]
	fn get_linebreak_property_latin1(codepoint: u8) -> u8 {
	let codepoint = codepoint as usize;
	UAX14_PROPERTY_TABLE[codepoint / 1024][(codepoint & 0x3ff)]
	}

Is this issue tracking the implementation of such an API?

[sffc]

Is this issue tracking the implementation of such an API?

Yes.

[aethanyc]

Does this issue depend on #883?

[sffc]

Does this issue depend on #883?

In part, but the non-binary enumerated property API you are requesting above needs additional work. In particular, it cannot be done until CodePointTrie is done.

[aethanyc]

Other than line breaker property, #943 also needs this API to map code point to various Unicode properties like Word_Break, Grapheme_Cluster_Break, etc.

[sffc]

I added a list of sub issues to the OP.

[sffc]

All six parts of this issue are done! Closing as fixed.