Commits on Apr 10, 2018

1. proc_macro: Avoid cached TokenStream more often …

   This commit adds even more pessimization to use the cached `TokenStream` inside
   of an AST node. As a reminder the `proc_macro` API requires taking an arbitrary
   AST node and transforming it back into a `TokenStream` to hand off to a
   procedural macro. Such functionality isn't actually implemented in rustc today,
   so the way `proc_macro` works today is that it stringifies an AST node and then
   reparses for a list of tokens.
   
   This strategy unfortunately loses all span information, so we try to avoid it
   whenever possible. Implemented in rust-lang#43230 some AST nodes have a `TokenStream`
   cache representing the tokens they were originally parsed from. This
   `TokenStream` cache, however, has turned out to not always reflect the current
   state of the item when it's being tokenized. For example `#[cfg]` processing or
   macro expansion could modify the state of an item. Consequently we've seen a
   number of bugs (rust-lang#48644 and rust-lang#49846) related to using this stale cache.
   
   This commit tweaks the usage of the cached `TokenStream` to compare it to our
   lossy stringification of the token stream. If the tokens that make up the cache
   and the stringified token stream are the same then we return the cached version
   (which has correct span information). If they differ, however, then we will
   return the stringified version as the cache has been invalidated and we just
   haven't figured that out.
   
   Closes rust-lang#48644
   Closes rust-lang#49846

   

   alexcrichton committed Apr 10, 2018
   Configuration menu

   • View commit details
   • Copy full SHA for 6d7cfd4
   • Browse repository at this point

   Copy the full SHA

   6d7cfd4 View commit details

   Browse the repository at this point in the history