parser: token -> normalized_token, nonnormalized_token -> token
#69384
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rust-highfive
added
the
S-waiting-on-review
Centril
reviewed
Feb 23, 2020
|
r=me with comment considered. |
petrochenkov
force-pushed
the
nounnorm
branch
from
8a5d59e
to
aa1e071
Compare
|
@bors r=Centril |
|
📌 Commit aa1e071deefb5a983bca3f9f0ff57779a5a8e7d1 has been approved by |
bors
added
S-waiting-on-bors
|
☔ The latest upstream changes (presumably #69393) made this pull request unmergeable. Please resolve the merge conflicts. |
bors
added
S-waiting-on-author
petrochenkov
force-pushed
the
nounnorm
branch
from
aa1e071
to
91971f8
Compare
|
@bors r=Centril |
|
📌 Commit 91971f8c6768eaee8212495db9d9b9a41788257f has been approved by |
bors
added
S-waiting-on-bors
|
@bors r- |
bors
added
S-waiting-on-author
petrochenkov
force-pushed
the
nounnorm
branch
from
91971f8
to
b2605c1
Compare
|
@bors r=Centril |
|
📌 Commit b2605c1 has been approved by |
|
🌲 The tree is currently closed for pull requests below priority 1000, this pull request will be tested once the tree is reopened |
bors
added
S-waiting-on-bors
bors
added a commit
that referenced
this pull request
Feb 28, 2020
Rollup of 10 pull requests Successful merges: - #68989 (Update RELEASES.md for 1.42.0) - #69340 (instantiate_value_path: on `SelfCtor`, avoid unconstrained tyvars) - #69384 (parser: `token` -> `normalized_token`, `nonnormalized_token` -> `token`) - #69452 (typeck: use `Pattern` obligation cause more for better diagnostics) - #69481 (use char instead of &str for single char patterns) - #69522 (error_derive_forbidden_on_non_adt: be more graceful) - #69538 (Stabilize `boxed_slice_try_from`) - #69539 (late resolve, visit_fn: bail early if there's no body.) - #69541 (Remove unneeded calls to format!()) - #69547 (remove redundant clones, references to operands, explicit boolean comparisons and filter(x).next() calls.) Failed merges: r? @ghost
This was referenced Feb 29, 2020
Dylan-DPC-zz
pushed a commit
to Dylan-DPC-zz/rust
that referenced
this pull request
Feb 29, 2020
rustc_parse: Tweak the function parameter name check The function doesn't need a full token, only its edition. Noticed while implementing rust-lang#69384. I'm still not sure whether normalized or unnormalized token is a better fit for the edition check here, so rust-lang#69384 and this PR just keep the status quo behavior. r? @Centril
JohnTitor
added a commit
to JohnTitor/rust
that referenced
this pull request
Mar 1, 2020
parser: Remove `Parser::prev_span` Follow-up to rust-lang#69384. r? @Centril
bors
added a commit
that referenced
this pull request
Mar 8, 2020
Centril
added a commit
to Centril/rust
that referenced
this pull request
Mar 9, 2020
rustc_parse: Remove `Parser::normalized(_prev)_token` Perform the "normalization" (renamed to "uninterpolation") on the fly when necessary. The final part of rust-lang#69579 rust-lang#69384 rust-lang#69376 rust-lang#69211 rust-lang#69034 rust-lang#69006. r? @Centril
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
So, after #69006, its follow-ups and an attempt to remove
Parser::prev_spanI came to the conclusion that the unnormalized token and its span is what you want in most cases, so it should be default.Normalization only makes difference in few cases where we are checking against
token::Identortoken::Lifetimespecifically.This PR uses
normalized_tokenfor those cases.Using normalization explicitly means that people writing code should remember about
NtIdentandNtLifetimein general. (That is alleviated by the fact thattoken.ident()andfn parse_ident_*are already written.)Remembering about
NtIdent, was, however, already the case, kind of, because the implicit normalization was performed only for the current/previous token, but not for things likelook_ahead.As a result, most of token classification methods in
token.rsalready takeNtIdentinto account (this PR fixes a few pre-existing minor mistakes though).The next step is removing
normalized(_prev)_tokenentirely and replacing it withtoken.ident()(mostly) andtoken.normalize()(occasionally).I want to make it a separate PR for that and run it though perf.
normalized_tokenfilled on every bump has both a potential to avoid repeated normalization, and to do unnecessary work in advance (it probably doesn't matter anyway, the normalization is very cheap).r? @Centril