i18n checklist for Handling base direction for strings

[takuki]

Handling base direction for strings

TD spec defines fields "description" and "title" in which the authors of a TD can put natural language text for humans to read. These are in a "default language" which can be explicitly defined in metadata. In addition "titles" and "descriptions" can be given which are sets of strings, each tagged with the language used. These can be used as alternatives or to replace the "title" and "description" data upon language negotiation (eg during HTTP requests).

Notes:

• The TD spec relies on JSON-LD 1.1, which supports language metadata but not direction metadata.
• When the JSON-LD standard changes to support direction metadata for strings, the support for direction will also be included in the TD spec. In the meantime we do not want to add our own metadata that might end up conflicting with the final JSON-LD 1.1 specification
• Our solution is to use the JSON-LD 1.1 @language mechanism to define a default language, and infer the base text direction from that language. We include assertions to require a script subtag when necessary to unambiguously specify the base text direction.
• We also provide MultiLanguage strings where each string has an associated language; these are used for "titles" and "descriptions"

The checklist for Handling base direction for strings is here.

1. ​Provide metadata constructs that can be used to indicate the base direction of any natural language string. more
   • JSON-LD 1.1 draft does not currently have a way to explicitly specify text direction
   • We do not want to add metadata that will conflict with a later JSON-LD 1.1 definition
2. ​Specify that consumers of strings should use heuristics, preferably based on the Unicode Standard first-strong algorithm, to detect the base direction of a string except where metadata is provided. more
   • We do NOT use or require strong-first as it is too fragile We actually included text for strong-first in the CR version
3. ​Where possible, define a field to indicate the default direction for all strings in a given resource or document. more
   • We cannot define an explicit field as it might conflict with a future JSON-LD 1.1 extension
4. ​Do NOT assume that a creating a document-level default without the ability to change direction for any string is sufficient. more
   • We do have a means to specify a default language, from which a base text direction can be derived and used as a starting point for processing strings
   • Within a string UTF-8 can be used to indicate changes in text direction
   • Strings in other languages than the base language can be specified in MultiLanguage objects
5. ​If metadata is not available due to legacy implementations and cannot otherwise be provided, specifications MAY allow a base direction to be interpolated from available language metadata. more
   • This is the solution we use.
   • We require a script subtag when it is necessary
6. ​Specifications MUST NOT require the production or use of paired bidi controls. more
   • We do not require this

[mmccool]

Personally, I think for consistency this should be handled by the JSON-LD 1.1 specification, NOT by the TD specification. If we add metadata for string direction we run the risk of being inconsistent with whatever JSON-LD 1.1 comes up with. If we add assertions wrt some of the other items (eg starting defaults) we run the risk of being incompatible with JSON-LD 1.1 assertions. Therefore, I think we should do nothing here, except (maybe) emphasizing that JSON-LD 1.1 rules for handling strings in various languages should be followed.

[mmccool]

See PR #620, which if accepted should clear up the issues around text direction for human-readable metadata. Unfortunately JSON-LD 1.1 is not (currently...) strict enough about this.

[mmccool]

I updated the description of this issue to reflect the current status of the spec; for example, we have removed "name" and "support" from the list of human-readable metadata. The checklist is a little odd since it gives a set of possible solutions, and we only have to implement one. I have checked the ones we actually use. The unchecked boxes reflect the solutions we did not use, and the associated comments explain why.

[mkovatsc]

@takuki Can we tick off the last two check boxes and close this issue?

• ​Providing metadata constructs indeed conflicts with JSON-LD compatibility, hence we have enough text on how to derive the base direction from the given language tags.
• The first-strong algorithm is now actually the main SHOULD-based rule, as it also is good for constrained devices that cannot keep an up-to-date database of language tags.

[mkovatsc]

@r12a please note that we use the "i18n-self-check" label for the I18N issues based on your checklist. ("i18n-comment" is for issues raised by your review, "i18n-tracking" was redundant).

[r12a]

@mkovatsc the i18n-tracking label is a W3C-wide horizontal tracker tag tied to our notification system. If that tag is present, we will be notified each time a new comment is added to the thread, when the issue is closed, etc.. Without it, we'll never notice that there are changes. I'm going to put it back here before sending this comment, so that our notification system picks up on #585 (comment), but it looks like you deleted the label entirely, so i'm a little worried about whether we will still get notifications for the several other issues we had labelled in this way. I guess we'll see.

[r12a]

Btw, for more information about the labels the i18n WG applies to other WG repos see https://www.w3.org/International/review-request#labels.

[r12a]

After our discussion during the i18n telecon on Thursday i reworked the recommendations in the string-meta document, to better show the relationships between the various approaches to supplying language/direction info. (This means that the checklist needs updating as soon as the dust settles.) See https://w3c.github.io/string-meta/#best-practices-recommendations-and-gaps This rework was prompted by the discussion we had.

I'd like to draw your attention to one part of that in particular, which says "Specify that, in the absence of other information, the default direction and default language are unknown."

I believe your spec says that the default for direction is LTR, in absence of other information. This seemed to us to be inappropriate, on reflection. Rather the direction should be set to unknown, so that first-strong heuristics will kick in. (see https://w3c.github.io/string-meta/#bp-not_only_default and the explanation just below)

I haven't had a chance to look yet. It may be that your wording around when FS heuristics should be used may avoid a problem here, but you may want to check that.

[mkovatsc]

I believe your spec says that the default for direction is LTR, in absence of other information

We fixed that after the call with you to:

If no language tag is given, the base direction SHOULD be inferred through first-strong heuristics or detection algorithms such as the CLDR Likely Subtags [LDML].

[mkovatsc]

i18n-tracking label is a W3C-wide horizontal tracker tag tied to our notification system

Sorry, my bad, I did not know about that. I will fix it in all related issues.

[mkovatsc]

Is the "i18n-comment" label enough or does it also need the "i18n-tracking"?

[r12a]

Only one of i18n-comment or i18n-tracking are needed. They're both linked into the notification system. We use i18n-comment for issues we raised or for other issues where we want to be asked if we are satisfied before transitioning. i18n-tracking is used for the other issues we just want to track.

[sebastiankb]

@aphillips provide us a detailed positive feedback about our I18N changes I think this issue can be closed

For more details, check https://github.com/w3c/wot-thing-description/issues?q=is%3Aissue+label%3Ai18n-comment+is%3Aclosed

Please reopen, if you see still any concerns.