Always capture tokens for macro_rules! arguments

[Aaron1011]

When we invoke a proc-macro, the TokenStream we pass to it may contain 'interpolated' AST fragments, represented by rustc_ast::token::Nonterminal. In order to correctly, pass a Nonterminal to a proc-macro, we need to have 'captured' its TokenStream at the time the AST was parsed.

Currently, we perform this capturing when attributes are present on items and expressions, since we will end up using a Nonterminal to pass the item/expr to any proc-macro attributes it is annotated with. However, Nonterminals are also introduced by the expansion of metavariables in macro_rules! macros. Since these metavariables may be passed to proc-macros, we need to have tokens available to avoid the need to pretty-print and reparse (see #43081).

This PR unconditionally performs token capturing for AST items and expressions that are passed to a macro_rules! invocation. We cannot know in advance if captured item/expr will be passed to proc-macro, so this is needed to ensure that tokens will always be available when they are needed.

This ensures that proc-macros will receive tokens with proper Spans (both location and hygiene) in more cases. Like all work on #43081, this will cause regressions in proc-macros that were relying on receiving tokens with dummy spans.

In this case, Crater revealed only one regression: the Pear crate (a helper for rocket), which was previously fixed as part of #73084.

This regression manifests itself as the following error:

[INFO] [stdout] error: proc macro panicked
[INFO] [stdout]    --> /opt/rustwide/cargo-home/registry/src/github.com-1ecc6299db9ec823/rocket_http-0.4.5/src/parse/uri/parser.rs:119:34
[INFO] [stdout]     |
[INFO] [stdout] 119 |             let path_and_query = pear_try!(path_and_query(is_pchar));
[INFO] [stdout]     |                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[INFO] [stdout]     |
[INFO] [stdout]     = help: message: called `Option::unwrap()` on a `None` value
[INFO] [stdout]     = note: this error originates in a macro (in Nightly builds, run with -Z macro-backtrace for more info)

It can be fixed by running cargo update -p pear, which updates your Cargo.lock to use the latest version of Pear (which includes a bugfix for the regression).

Split out from #73084

[rust-highfive]

r? @estebank

(rust_highfive has picked a reviewer for you, use r? to override)

[Aaron1011]

r? @petrochenkov

[Aaron1011]

@petrochenkov: I've addressed your comments.

[petrochenkov]

LGTM.

I don't expect breakage, but there may be surprises.
Let's run this through crater.
@bors try

[bors]

⌛ Trying commit 74599cd with merge 8bd52ddba12ca76b95d401142c1bd190bda1fd2a...

[bors]

💔 Test failed - checks-azure

[Aaron1011]

@bors try

[bors]

⌛ Trying commit 74599cd with merge 6213129f69f3dab2041bcf4ba3c00595328028c0...

[nnethercote]

This was a perf loss on the artificial deep-vector benchmark, which probably doesn't matter much.

[francois-blanchard]

I had the same problem during the rocket compilation.
cargo update -p pear and cargo update -p pear_codegen are fixed my issue.
Thx @Aaron1011.

Rust version

$ rustc --version
rustc 1.46.0-nightly (23744c84d 2020-07-14)

Cargo.toml

[dependencies]
rocket = "0.4.5"
...
pear = "0.1.4"
pear_codegen = "0.1.4"