Allow non-ASCII identifiers #2457
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,161 @@ | ||
| - Feature Name: unicode_idents | ||
| - Start Date: 2018-06-03 | ||
| - RFC PR: (leave this empty) | ||
| - Rust Issue: (leave this empty) | ||
|
|
||
| # Summary | ||
| [summary]: #summary | ||
|
|
||
| Allow non-ASCII letters (such as accented characters, Cyrillic, Greek, Kanji, etc.) in Rust identifiers. | ||
|
|
||
| # Motivation | ||
| [motivation]: #motivation | ||
|
|
||
| Rust is written by many people who are not fluent in the English language. Using identifiers in ones native language eases writing and reading code for these developers. | ||
|
|
||
| The rationale from [PEP 3131] nicely explains it: | ||
|
|
||
| > ~~Python~~ *Rust* code is written by many people in the world who are not familiar with the English language, or even well-acquainted with the Latin writing system. Such developers often desire to define classes and functions with names in their native languages, rather than having to come up with an (often incorrect) English translation of the concept they want to name. By using identifiers in their native language, code clarity and maintainability of the code among speakers of that language improves. | ||
| > | ||
| > For some languages, common transliteration systems exist (in particular, for the Latin-based writing systems). For other languages, users have larger difficulties to use Latin to write their native words. | ||
|
|
||
| Additionally some math oriented projects may want to use identifiers closely resembling mathematical writing. | ||
|
There was a problem hiding this comment. NFKC removes the mathematical distinction of font style, as in ⟨ℍ⟩ vs. ⟨ℋ⟩. There was a problem hiding this comment. @dscorbett yeah the idea is that while you can use stylized characters you should not go completely overboard and use all of them in the same formula. This applies also to the distinguished in natural language orthographies. |
||
|
|
||
| # Guide-level explanation | ||
| [guide-level-explanation]: #guide-level-explanation | ||
|
|
||
| Identifiers include variable names, function and trait names and module names. They start with a letter or an underscore and may be followed by more letters, digits and some connecting punctuation. | ||
|
|
||
| Examples of valid identifiers are: | ||
|
|
||
| * ASCII letters and digits: `image_width`, `line2`, `Photo`, `el_tren`, `_unused` | ||
| * words containing accented characters: `garçon`, `hühnervögel` | ||
| * identifiers in other scripts: `Москва`, `東京`, ... | ||
|
|
||
| Examples of invalid identifiers are: | ||
|
|
||
| * Keywords: `impl`, `fn`, `_` (underscore), ... | ||
| * Identifiers starting with numbers or containing "non letters": `42_the_answer`, `third√of7`, `◆◆◆`, ... | ||
|
There was a problem hiding this comment. I would actually really like it if the √ symbol would be allowed as well (e.g. to white list it). In academic formula context this can make formulas in code even closer to the formulas from papers and thus easier to read in the academic context (I've actively missed this symbol when doing the same with Python 3). As an example I take the right top most formal from this picture: let √ = f64::sqrt;
let σ² = σ.powi(2);
let exp = f64::exp;
let result = 1/√(2*π*σ²) * exp(-(1/2) * ((x - x_abs)/σ))instead of let sqrt = f64::sqrt;
let σ² = σ.powi(2);
let exp = f64::exp;
let result = 1/sqrt(2*π*σ²) * exp(-(1/2) * ((x - x_abs)/σ))and also compare it to the most easy to read one I could think of without even using greek symbols: let sqrt = f64::sqrt;
let sigma2 = sigma.powi(2);
let exp = f64::exp;
let result = 1/sqrt(2*pi*sigma2) * exp(-(1/2) * ((x - x_abs)/sigma))There was a problem hiding this comment.
I am sympathetic to this idea but would really want to avoid to diverge from UAX31 for a single character (or a few). I would prefer if someone created a "custom operators" RFC that allowed to define additional operators. This would enable the |
||
| * Emojis: 🙂, 🦀, 💩, ... | ||
|
There was a problem hiding this comment. Bear in mind, lots of random things are RGI emoji in unicode, there's no single "category" for this. There was a problem hiding this comment. Indeed, the emoji ⟨ℹ️⟩ would be a valid identifier. |
||
|
|
||
| Similar Unicode identifiers are normalized: `a1` and `a₁` (a<subscript 1>) refer to the same variable. This also applies to accented characters which can be represented in different ways. | ||
|
|
||
| To disallow any Unicode identifiers in a project (for example to ease collaboration or for security reasons) limiting the accepted identifiers to ASCII add this lint to the `lib.rs` or `main.rs` file of your project: | ||
|
|
||
| ```rust | ||
| #![forbid(unicode_idents)] | ||
|
There was a problem hiding this comment. To split hairs: this should be Rust source files are always Unicode. The U+0061 to U+007A range is part of Unicode. |
||
| ``` | ||
|
|
||
| Some Unicode character look confusingly similar to each other or even identical like the Latin **A** and the Cyrillic **А**. The compiler may warn you about easy to confuse names in the same scope. If needed (but not recommended) this warning can be silenced with a `#[allow(confusable_unicode_idents)]` annotation on the enclosing function or module. | ||
|
|
||
| ## Usage notes | ||
|
|
||
| All code written in the Rust Language Organization (*rustc*, tools, std, common crates) will continue to only use ASCII identifiers and the English language. | ||
|
|
||
| For open source crates it is suggested to write them in English and use ASCII-only. An exception can be made if the application domain (e.g. math) benefits from Unicode and the target audience (e.g. for a crate interfacing with Russian passports) is comfortable with the used language and characters. Additionally crates should consider to provide an ASCII-only API. | ||
|
|
||
| Private projects can use any script and language the developer(s) desire. It is still a good idea (as with any language feature) not to overdo it. | ||
|
|
||
| # Reference-level explanation | ||
| [reference-level-explanation]: #reference-level-explanation | ||
|
|
||
| Identifiers in Rust are based on the [Unicode® Standard Annex #31 Unicode Identifier and Pattern Syntax][TR31]. Rust compilers shall use at least Revision 27 of the standard. | ||
|
There was a problem hiding this comment. The revision should be specified exactly. Otherwise, the same identifier in the same Rust version could be valid in one compiler but another. There was a problem hiding this comment. I don't want to lock Rust compilers to a specific Unicode revision. An alternative would be to state the supported Unicode revision for each Rust edition. There was a problem hiding this comment. New versions of Rust should use new versions of Unicode, of course, but any given version of Rust should have an unambiguous definition of identifiers. Your alternative is good. Here is why the current text is wrong. Let’s say, for example, that revision 28 adds ⟨⍙⟩ to XID_Start, just in time for Rust 1.40.0. Would ⟨⍙abc⟩ be a valid identifier in Rust 1.40.0? It would be valid in compilers that use revision 28, but not in compilers that still use revision 27. Both compilers would be correct and yet disagree about something as basic as identifier validity. There was a problem hiding this comment. FWIW we’ve already been updating Unicode for the standard library (e.g. There was a problem hiding this comment. UAX 31 should be referenced for information/context, but I think that its revision number can be removed entirely. This RFC defines identifier syntax based on the To maintain Rust’s stability promise, we need to ensure that:
My reading of UAX 31 and 15 is that we can do that and still update the to newer versions of Unicode in newer versions of Rust backward-compatibly without involving Rust editions. We should have some way communicate to users the version mapping, and I think that the existing This translates 1. above into the requirement UAX31-R1b Stable Identifier:
Per https://www.unicode.org/reports/tr31/#Backward_Compatibility I believe that Additionally, code points that are not assigned in a given Unicode version cannot be in
Updating to a newer Unicode does not change the result of
It would a Unicode version rather than in a revision of UAX 31 that adds it, but yes, if Rust 1.40.0 is the one that upgrades to that version of Unicode (regardless of when that version of Unicode was released), then There was a problem hiding this comment. Right. It was a rhetorical question to show why specifying the Unicode version exactly is necessary. There was a problem hiding this comment. Bear in mind, the unicode annexes also do get updated every few versions, often to accommodate for new kinds of code points but sometimes to just fix things. |
||
|
|
||
| The lexer defines identifiers as: | ||
|
|
||
| > **<sup>Lexer:<sup>** | ||
| > IDENTIFIER_OR_KEYWORD: | ||
| > XID_Start XID_Continue<sup>\*</sup> | ||
|
There was a problem hiding this comment. How does this compare to Go's identifier grammar: https://golang.org/ref/spec#Identifiers? It doesn't use XID_Start and XID_Continue, but instead the "Letter" and "Number, decimal digit" character classes. There was a problem hiding this comment. XID_Start includes "Letter" but also includes "Number, Letter" (for example Roman numerals like Ⅻ) and a few characters for compatibility and to ensure that XID_Start is closed under NFKC. XID_Continue includes all of XID_Start and "Number, decimal digit" , marks to build combining characters from parts ( There is golang/go#194 where combining characters are discussed as they are needed to write some Asian languages. See also: https://golang.org/doc/faq#unicode_identifiers |
||
| > | `_` XID_Continue<sup>+</sup> | ||
|
There was a problem hiding this comment. The guide-level explanation says that identifiers may contain “more letters, digits and some connecting punctuation”, but XID_Continue does not include connecting punctuation. There was a problem hiding this comment. XID_Continue includes characters from the Pc class called "Punctuation, Connector". There was a problem hiding this comment. So it does. I must have been looking at XID_Start. |
||
| > | ||
| > IDENTIFIER : | ||
| > IDENTIFIER_OR_KEYWORD <sub>*Except a [strict] or [reserved] keyword*</sub> | ||
|
|
||
| `XID_Start` and `XID_Continue` are used as defined in the aforementioned standard. The definition of identifiers is forward compatible with each successive release of Unicode as only appropriate new characters are added to the classes but none are removed. | ||
|
|
||
| Two identifiers X, Y are considered to be equal if their [NFKC forms][TR15] are equal: NFKC(X) = NFKC(Y). | ||
|
|
||
| A `unicode_idents` lint is added to the compiler. This lint is `allow` by default. The lint checks if any identifier in the current context contains a codepoint with a value equal to or greater than 0x80 (outside ASCII range). Not only locally defined identifiers are checked but also those imported from other crates and modules into the current context. | ||
|
There was a problem hiding this comment. I don't think the lint should be As for checking imported idents as well, I think this should be a separate lint. You might want to be able to import something from a foreign language crate but not want to have foreign language idents in your own code. There was a problem hiding this comment. Reposting my comment from #2455 (comment), I think that there should be different lints, with different levels, as some languages should be allow/warn by default, and some others should be deny by default. |
||
|
|
||
| ## Confusable detection | ||
|
|
||
| Rust compilers should detect confusingly similar Unicode identifiers and warn the user about it. | ||
|
|
||
| Note: This is *not* a mandatory for all Rust compilers as it requires considerable implementation effort and is not related to the core function of the compiler. It rather is a tool to detect accidental misspellings and intentional homograph attacks. | ||
|
|
||
| A new `confusable_unicode_idents` lint is added to the compiler. The default setting is `warn`. | ||
|
|
||
| Note: The confusable detection is set to `warn` instead of `deny` to enable forward compatibility. The list of confusable characters will be extended in the future and programs that were once valid would fail to compile. | ||
|
|
||
| The confusable detection algorithm is based on [Unicode® Technical Standard #39 Unicode Security Mechanisms Section 4 Confusable Detection][TR39Confusable]. For every distinct identifier X in the current scope execute the function `skeleton(X)`. If there exist two distinct identifiers X and Yin the same crate where `skeleton(X) = skeleton(Y)` report it. | ||
|
There was a problem hiding this comment.
There was a problem hiding this comment. I see why this would be a good idea but if we decide to do it I would like to consult with the Unicode people first to find out why they did not do it in this way. There was a problem hiding this comment. I suspect they didn’t do it that way because UTS #39’s confusability data’s “primary goal is to include characters that would be Status=Allowed”, and default-ignorable code points are Status=Restricted. There was a problem hiding this comment. I kinda want to get someone from the compiler team to approve that this is something that's definitely doable and won't adversely affect performance. This seems tricky to get right. There was a problem hiding this comment. I think we already have some logic like this but we only use it in the lexer when we see a character we don't know. Assuming running But that's only if the check is crate-wide with no scoping taking into account. There was a problem hiding this comment. Yeah the current proposal involves scoping which I'm concerned about. There was a problem hiding this comment. "If there exist two distinct identifiers X and Y in the same crate where" seems to suggest scoping isn't involved? There was a problem hiding this comment. Elsewhere: "The compiler may warn you about easy to confuse names in the same scope." There's also the line before this one, which mentions scope. @pyfisch, what did you intend here? There was a problem hiding this comment. Its a mistake that "scope" is mentioned in the first sentence. The skeleton for each identifier should be computed and all identifiers in a crate should be compared against each other. I assume this is easiest with a hashmap as suggested by @eddyb. Having to build just one hashmap for the whole crate should be faster to execute and easier to code than building one for each scope. (Correct me if I am wrong.) There was a problem hiding this comment. I think this should read: Report a warning if there exist two distinct keywords or identifiers We'd forbid many identifiers as homographs of identifiers importable from There was a problem hiding this comment. I think the "importable into" bit partially resolves the issues for which I suggested explicitly declaring scripts. |
||
|
|
||
| # Drawbacks | ||
| [drawbacks]: #drawbacks | ||
|
|
||
| * "ASCII is enough for anyone." As source code should be written in English and in English only (source: various people) no charactes outside the ASCII range are needed to express identifiers. Therefore support for Unicode identifiers introduces unnecceray complexity to the compiler. | ||
|
There was a problem hiding this comment. "source code should be written in English and in English only" is not an argument, it's an opinion - one that is easily shown to be partial by pointing out how ballistic people can go when e.g. variable names don't convey the correct meaning of the variable they name; now think if, because you're not a good English speaker, they convey no meaning to you at all... My point being that obviously English speakers see no problem whatsoever with using English variable names. There was a problem hiding this comment. @kimhyunkang exposes even more reasons why this stance is terrible for diversity/inclusion in #2457 (comment) There was a problem hiding this comment. Also keep in mind that especially for younger programmers (whose native language is not English) this doesn't hold. People try out programming while they are in school and have courses on English but aren't nearly fluent yet. It's much easier to write code in your native language at that point in life, so you can focus on learning how to code instead of also needing to learn English at the same time. Source: I'm Dutch and although all code I write now is English I have definitely written Dutch code in my younger years when I wasn't fluent enough in English yet and was learning how to code. Luckily I wasn't held back by non ASCII identifiers, because There was a problem hiding this comment.
That's true. But I still still hear this argument often. The underlying argument is that one needs to write in English so other people can read it. And even if you write code just for yourself you should write in English to learn it. If you think this statement should be rephrased please suggest something else. btw I had a similar experience as @JelteF There was a problem hiding this comment.
Then the underlying argument is proved false by the existence and size of huge Chinese communities where English is definitely not the primary way of expressing yourself, nor it is required to look up information. Don't get me wrong: this obviously holds true if you're in an English-speaking community, country or organization. What I'm arguing is that the converse is not true.
I think this corollary is preposterous because it boils down to: "since somebody arbitrarily decided for convenience that everybody else should write code in English, you have to learn English even if you are the only person you will ever read the code you write". To see how tone deaf this argument is, replace "code" with any of "letter", "notes", "book" or "song" and then tell me if it makes sense... |
||
| * "Foreign characters are hard to type." Usually computer keyboards provide access to the US-ASCII printable characters and the local language characters. Characters from other scripts are difficult to type, require entering numeric codes or are not available at all. These characters either need to be copy-pasted or entered with an alternative input method. | ||
| * "Foreign characters are hard to read." If one is not familiar with the characters used it can be hard to tell them apart (e.g. φ and ψ) and one may not be able refer to the identifiers in an appropriate way (e.g. "loop" and "trident" instead of phi and psi) | ||
| * "My favorite terminal/text editor/web browser" has incomplete Unicode support." Even in 2018 some characters are not widely supported in all places where source code is usually displayed. | ||
| * Homoglyph attacks are possible. Without confusable detection identifiers can be distinct for the compiler but visually the same. Even with confusable detection there are still similar looking characters that may be confused by the casual reader. | ||
|
|
||
| # Rationale and alternatives | ||
| [alternatives]: #alternatives | ||
|
|
||
| As stated in [Motivation](#motivation) allowing Unicode identifiers outside the ASCII range improves Rusts accessibility for developers not working in English. Especially in teaching and when the application domain vocabulary is not in English it can be beneficial to use names from the native language. To facilitate this it is necessary to allow a wide range of Unicode character in identifiers. The proposed implementation based on the Unicode TR31 is already used by other programming languages (e.g. Python 3) and is implemented behind the `non_ascii_idents` in *rustc* but lacks the NFKC normalization proposed. | ||
|
|
||
| Possible variants: | ||
|
|
||
| 1. Require all identifiers to be in NFKC or NFC form. | ||
| 2. Two identifiers are only equal if their codepoints are equal. | ||
| 3. Perform NFC mapping instead of NFKC mapping for identifiers. | ||
| 4. Only a number of common scripts could be supported. | ||
|
There was a problem hiding this comment. Other possible variants are the restriction levels of UTS #39. There was a problem hiding this comment. This is listed as variant 5. |
||
| 5. A [restriction level][TR39Restriction] is specified allowing only a subset of scripts and limit script-mixing within an identifier. | ||
|
|
||
| An alternative design would use [Immutable Identifiers][TR31Alternative] as done in [C++]. In this case a list of Unicode codepoints is reserved for syntax (ASCII operators, braces, whitespace) and all other codepoints (including currently unassigned codepoints) are allowed in identifiers. The advantages are that the compiler does not need to know the Unicode character classes XID_Start and XID_Continue for each character and that the set of allowed identifiers never changes. It is disadvantageous that all not explicitly excluded characters at the time of creation can be used in identifiers. This allows developers to create identifiers that can't be recognized as such. It also impedes other uses of Unicode in Rust syntax like custom operators if they were not initially reserved. | ||
|
|
||
| It always a possibility to do nothing and limit identifiers to ASCII. | ||
|
|
||
| It has been suggested that Unicode identifiers should be opt-in instead of opt-out. The proposal chooses opt-out to benefit the international Rust community. New Rust users should not need to search for the configuration option they may not even know exists. Additionally it simplifies tutorials in other languages as they can omit an annotation in every code snippet. | ||
|
|
||
| ## Confusable detection | ||
|
|
||
| The current design was chosen because the algorithm and list of similar characters are already provided by the Unicode Consortium. A different algorithm and list of characters could be created. I am not aware of any other programming language implementing confusable detection. The confusable detection was primarily included because homoglyph attacks are a huge concern for some member of the community. | ||
|
|
||
| Instead of offering confusable detection the lint `forbid(unicode_idents)` is sufficient to protect project written in English from homoglyph attacks. Projects using different languages are probably either written by students, by a small group or inside a regional company. These projects are not threatened as much as large open source projects by homoglyph attacks but still benefit from the easier debugging of typos. | ||
|
|
||
| # Prior art | ||
| [prior-art]: #prior-art | ||
|
|
||
| "[Python PEP 3131][PEP 3131]: Supporting Non-ASCII Identifiers" is the Python equivalent to this proposal. The proposed identifier grammar **XID_Start XID_Continue<sup>\*</sup>** is identical to the one used in Python 3. | ||
|
|
||
| [JavaScript] supports Unicode identifiers based on the same Default Identifier Syntax but does not apply normalization. | ||
|
|
||
| The [CPP reference][C++] describes the allowed Unicode identifiers it is based on the immutable identifier principle. | ||
|
|
||
| [Java] also supports Unicode identifiers. Character must belong to a number of Unicode character classes similar to XID_start and XID_continue used in Python. Unlike in Python no normalization is performed. | ||
|
|
||
| The [Go language][Go] allows identifiers in the form **Letter (Letter | Number)\*** where **Letter** is a Unicode letter and **Number** is a Unicode decimal number. This is more restricted than the proposed design mainly as is does not allow combining characters needed to write some languages such as Hindi. | ||
|
|
||
| # Unresolved questions | ||
| [unresolved]: #unresolved-questions | ||
|
|
||
| * Which context is adequate for confusable detection: file, current scope, crate? | ||
| * Are Unicode characters allowed in `no_mangle` and `extern fn`s? | ||
| * How do Unicode names interact with the file system? | ||
| * Are crates with Unicode names allowed and can they be published to crates.io? | ||
| * Are `unicode_idents` and `confusable_unicode_idents` good names? | ||
| * Should [ZWNJ and ZWJ be allowed in identifiers][TR31Layout]? | ||
| * Should *rustc* accept files in a different encoding than *UTF-8*? | ||
|
There was a problem hiding this comment. This one is easy: no. Why would it? UTF-8 by definition supports all of Unicode. Also, source files encodings seem off-topic for this RFC. |
||
|
|
||
| [PEP 3131]: https://www.python.org/dev/peps/pep-3131/ | ||
| [TR15]: https://www.unicode.org/reports/tr15/ | ||
| [TR31]: http://www.unicode.org/reports/tr31/ | ||
| [TR31Alternative]: http://unicode.org/reports/tr31/#Alternative_Identifier_Syntax | ||
| [TR31Layout]: https://www.unicode.org/reports/tr31/#Layout_and_Format_Control_Characters | ||
| [TR39Confusable]: https://www.unicode.org/reports/tr39/#Confusable_Detection | ||
| [TR39Restriction]: https://www.unicode.org/reports/tr39/#Restriction_Level_Detection | ||
| [C++]: https://en.cppreference.com/w/cpp/language/identifiers | ||
| [Julia Unicode PR]: https://github.com/JuliaLang/julia/pull/19464 | ||
| [Java]: https://docs.oracle.com/javase/specs/jls/se10/html/jls-3.html#jls-3.8 | ||
| [JavaScript]: http://www.ecma-international.org/ecma-262/6.0/#sec-names-and-keywords | ||
| [Go]: https://golang.org/ref/spec#Identifiers | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Some languages use U+200C ZERO WIDTH NON-JOINER and U+200D ZERO WIDTH JOINER, which are not in XID_Continue. Should they be allowed too? See section 2.3 of UAX #31.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To have proper support then they should be allowed, as it affects the rendering of text, which may cause it to have different meanings (I'm not sure, but it seems likely to me). However, this can cause issues like in C++ and Swift for example， different number of zero-width joiners or spaces are different identifiers, and that leads to a readability nightmare.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That is why UAX #31 suggests allowing them only in certain positions, to avoid having multiple distinct identifiers look identical.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So AIUI these are used for:
Forcing explicit viramas in Indic scripts in consonant clusters. There is a semantic difference but it's exceedingly minor and mostly comes in play for Sanskrit. It's a pretty minor difference; AIUI it's used in certain kinds of compound words and is very easily omitted
Certain vowel presentational forms in Bengali and Oriya. Also minor.
Forcing letters to take different (word-medial, etc) forms in the Perso-Arabic script, used for:
Arabic affixes, when shown in isolation or when used with non-arabic words. Imagine you had to write something like "Rust's" where "Rust" is in the Latin script but the 's is in Arabic, you need the Arabic suffix to not use a word-initial text form. An example of this is the ب prefix preposition
Arabic abbreviations
AFAICT these are all somewhat optional (though preferred). The abbreviations one might be the most used.
(that said, there are much larger problems with using an RTL script in rust)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ZWNJ has important use in Persian at least https://en.m.wikipedia.org/wiki/Zero-width_non-joiner