#[derive(...)] doesn't invalidate tokens for nested items

[petrochenkov]

Reproduction:

// check-pass
// aux-build:test-macros.rs

#![feature(stmt_expr_attributes)]
#![feature(proc_macro_hygiene)]

#[macro_use]
extern crate test_macros;

#[derive(PartialEq, Eq)]
struct S {
    field: [u8; #[print_attr] {
        #[cfg(FALSE)] { 10 }
        #[cfg(not(FALSE))]  { 11 }
    }],
}

fn main() {}

#[derive] will fully configure the item S and remove #[cfg(FALSE)] { 10 } before outputting the item or passing it to PartialEq and Eq.
#[print_attr] however will print tokens for both cfg cases because its input tokens are not invalidated and are stale.

Fix:

• #[derive] should walk the whole input item and invalidate tokens for all nested nodes, not only for the item itself.
• Attribute macros should be ready to face input with no tokens and use CanSynthesizeMissingTokens::Yes.

cc @Aaron1011

[petrochenkov]

The part about CanSynthesizeMissingTokens::Yes also becomes relevant with #79078, because once that PR lands #[my_attr] in

#[derive(MyTrait)]
#[my_attr]
struct S { ... }

should be ready to receive an item with invalidated tokens as an input.

[petrochenkov]

I'm going to feature-gate macro attributes on nested nodes in #79078 because they are very similar to attribute macros after derive in the sense that they allow macros to observe derive's output, and the lang team's position is that it shouldn't be allowed on stable yet.
(It's unlikely to be used in practice, but beta crater runs will us is something breaks.)

[Aaron1011]

#80689 will automatically fix the #[derive(MyTrait)] #[my_attr] case, since the AST node for the derive target will have its tokens cfg-stripped, and then passed to #[my_attr]. It will not directly fix the 'nested attribute' case, but will give us the infrastructure to do so (we'll want to run cfg-stripping for the tokens of all AST nodes, not just the top-level derive target).

[petrochenkov]

#[cfg_eval] implemented in #82682 is also affected by this issue.

[petrochenkov]

@nnethercote This was probably addressed by your recent fixes to token collection?

[nnethercote]

Has the behaviour changed? If so, I guess those fixes are responsible.

[petrochenkov]

Yes, the behavior changed, I've added a test in #137544.
(The language behavior here changed as well, in #87220.)