diff --git a/src/tokens.md b/src/tokens.md index 631b0fcdb..fc401427f 100644 --- a/src/tokens.md +++ b/src/tokens.md @@ -92,7 +92,7 @@ A suffix is a non-raw identifier immediately (without whitespace) following the primary part of a literal. Any kind of literal (string, integer, etc) with any suffix is valid as a token, -and can be passed to a macro without producing an error. +and can be passed to a macro without producing an error. The macro itself will decide how to interpret such a token and whether to produce an error or not. ```rust @@ -101,7 +101,7 @@ macro_rules! blackhole { ($tt:tt) => () } blackhole!("string"suffix); // OK ``` -However, suffixes on literal tokens parsed as Rust code are restricted. +However, suffixes on literal tokens parsed as Rust code are restricted. Any suffixes are rejected on non-numeric literal tokens, and numeric literal tokens are accepted only with suffixes from the list below. @@ -150,10 +150,11 @@ A _string literal_ is a sequence of any Unicode characters enclosed within two `U+0022` (double-quote) characters, with the exception of `U+0022` itself, which must be _escaped_ by a preceding `U+005C` character (`\`). -Line-break characters are allowed in string literals. Normally they represent -themselves (i.e. no translation), but as a special exception, when an unescaped -`U+005C` character (`\`) occurs immediately before the newline (`U+000A`), the -`U+005C` character, the newline, and all whitespace at the beginning of the +Line-breaks are allowed in string literals. A line-break is either a newline +(`U+000A`) or a pair of carriage return and newline (`U+000D`, `U+000A`). Both +flavors are normally translated to `U+000A`, but as a special exception, when an +unescaped `U+005C` character (`\`) occurs immediately before the line-break, the +`U+005C` character, the line-break, and all whitespace at the beginning of the next line are ignored. Thus `a` and `b` are equal: ```rust