spec: bytes data type

[alandonovan]

This change adds initial specification of the bytes data type
following length discussion in #112.
It also explains the implementation-dependent encoding of
text strings, and the \u \U \X escapes.

More will follow, but let's get the easy parts out of the way first.

Updates #112

[brandjon]

(Still reviewing)

[brandjon]

Github won't let me comment on the above lines so quoting here:

Regardless of the platform's convention for text line endings---for example, a linefeed (\n) on UNIX, or a carriage return followed by a linefeed (\r\n) on Microsoft Windows---an unescaped line ending in a multiline string literal always denotes a line feed (\n).

Do we specify what constitutes a line ending in a Starlark source file? Specifically, what the algorithm is for converting a raw U+000D or U+000A or a pair of them, into a single U+000A?

Starlark also supports raw string literals, which look like an ordinary single- or double-quotation preceded by r. Within a raw string literal, there is no special processing of backslash escapes, other than an escaped quotation mark (which denotes a literal quotation mark), or an escaped newline (which denotes a backslash followed by a newline). This form of quotation is typically used when writing strings that contain many quotation marks or backslashes (such as regular expressions or shell commands) to reduce the burden of escaping:

r'\'' denotes a literal backslash and quote, not a quote by itself. Also, raw string literals don't help you when there are many literal quotes, since you still have to escape them (if they match the string's opening and closing quote type), but they do help you when there are many literal backslashes.

[alandonovan]

Do we specify what constitutes a line ending in a Starlark source file? Specifically, what the algorithm is for converting a raw U+000D or U+000A or a pair of them, into a single U+000A?

What more needs to be said? The scanner needs to recognize line endings (however they are defined by the platform), in three places:

1. escaped, in which case they are ignored;
2. unescaped, outside a string literal, where they make a NEWLINE token;
3. unescaped, in a multiline string literal, where they make a \n (as the quoted paragraph explains).

[brandjon]

The scanner is part of the spec, no? I believe Python defines newlines in a platform-independent way, and we should probably do the same. A raw \r\n on unix should still produce a single \n, not a \r\n, inside a multiline string literal.

[brandjon]

A TODO is fine for unblocking this PR.

[alandonovan]

My point is that the only place that needs to take a stance on the concrete representation of a line ending is case 3, which already spells it out thus:

Regardless of the platform's convention for text line endings---for
example, a linefeed (\n) on UNIX, or a carriage return followed by a
linefeed (\r\n) on Microsoft Windows---an unescaped line ending in a
multiline string literal always denotes a line feed (\n).

[brandjon]

Driveby: "type(x) returns a string describing the type of its operand." -> backticks around type(x).

[brandjon]

Also:

Operand = identifier
        | int | float | string
        | ListExpr | ListComp
        | DictExpr | DictComp
        | '(' [Expression [',']] ')'

Needs a | bytes there.

[alandonovan]

Thanks Jon; PTAL.

[brandjon]

The scanner is part of the spec, no? I believe Python defines newlines in a platform-independent way, and we should probably do the same. A raw \r\n on unix should still produce a single \n, not a \r\n, inside a multiline string literal.

[alandonovan]

Driveby: "type(x) returns a string describing the type of its operand." -> backticks around type(x).

Done.

Needs a | bytes there.

Done.

The scanner is part of the spec, no? I believe Python defines newlines in a platform-independent way, and we should probably do the same. A raw \r\n on unix should still produce a single \n, not a \r\n, inside a multiline string literal.

Yes. We say that here:
"""
Regardless of the platform's convention for text line endings---for
example, a linefeed (\n) on UNIX, or a carriage return followed by a
linefeed (\r\n) on Microsoft Windows---an unescaped line ending in a
multiline string literal always denotes a line feed (\n).
"""