Lexer changes #15339
Merged
Lexer changes #15339
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
re-r? last two commits are new. @brson @jbclements |
|
Rebasing those doc comments changes must be painful. Why not make those a separate pull request, so we can just get those out of the way? EDIT: nvm, pretty much good to go here, pending q. about lifetimes |
|
Ok, all done. Last commit is new, and completes RFC 21. Still an open question about whether 011b0fb is OK. |
|
I'm inclined to say it's fine, since we do the same thing elsewhere. It's also wonderful shorthand to have... |
| @@ -362,28 +374,34 @@ impl<'a> StringReader<'a> { | |||
| } | |||
| self.bump(); | |||
| } | |||
| let ret = self.with_str_from(start_bpos, |string| { | |||
| return self.with_str_from(start_bpos, |string| { | |||
| let tok; | |||
|
r=me with the 3 nits. |
|
+9000 very timely for rustfmt (re: lexing comments / ws) |
Now that the lexer is more robust, these tests don't need to be in separate files. Yay!
Rather than just dumping the id in the interner, which is useless, actually print the interned string. Adjust the lexer logging to use Show instead of Poly.
This is technically unsafe but interned strings are considered immortal.
This shuffles things around a bit so that LIT_CHAR and co store an Ident
which is the original, unaltered literal in the source. When creating the AST,
unescape and postprocess them.
This changes how syntax extensions can work, slightly, but otherwise poses no
visible changes. To get a useful value out of one of these tokens, call
`parse::{char_lit, byte_lit, bin_lit, str_lit}`
[breaking-change]
This removes a bunch of token types. Tokens now store the original, unaltered numeric literal (that is still checked for correctness), which is parsed into an actual number later, as needed, when creating the AST. This can change how syntax extensions work, but otherwise poses no visible changes. [breaking-change]
Now, the lexer will categorize every byte in its input according to the grammar. The parser skips over these while parsing, thus avoiding their presence in the input to syntax extensions.
bors
added a commit
that referenced
this pull request
Jul 9, 2014
Mostly minor things that rebasing is becoming painful.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Mostly minor things that rebasing is becoming painful.