Tracking issue: declarative macros 2.0

[nrc]

Tracking issue for declarative macros 2.0 (aka macro aka decl_macro aka macros-by-example).

RFC: https://github.com/rust-lang/rfcs/blob/master/text/1584-macros.md

RFC PR: rust-lang/rfcs#1584

cc @rust-lang/compiler

---

About tracking issues

Tracking issues are used to record the overall progress of implementation.
They are also used as hubs connecting to other relevant issues, e.g., bugs or open design questions.
A tracking issue is however not meant for large scale discussion, questions, or bug reports about a feature.
Instead, open a dedicated issue for the specific matter and add the relevant feature gate label.
Discussion comments will get marked as off-topic or deleted.
Repeated discussions on the tracking issue may lead to the tracking issue getting locked.

Tasks

• Complete the draft hygiene prototype and RFC.
  • Hygiene for items, explicit imports, lexical scopes, and module scopes.
  • Hygiene for globs and trait methods (w.r.t. extension trait candidates in scope for a method call).
  • Hygiene for type directed name resolutions (i.e. methods, associated types, and fields).
  • Support macro in blocks as well as modules.
  • Implement inter-crate hygiene, except for nested macros.
  • Implement "hygiene bending" for when users want a name from macro def to "escape".
    • Examples: New declarative macros, functions and fields not being recognized #91249, Declaring modules inside decl macros 2.0 #46342
  • Implement inter-crate hygiene for nested macros (pending macro def encoding).
  • Make private_in_public hygienic?
  • Make unsafe and lints hygienic (if appropriate)?
• Add variant ast::ItemKind::MacroDef for macro_rules! items and macro items (PR syntax: add ast::ItemKind::MacroDef, simplify hygiene info #40220).
• Encode macros in the crate metadata using TokenStream, not String.
• Fix Inferred types should be checked for privacy violations #30476 (private_in_public details).
• Fix span issues If an error is not affected by a macro, don't print macro notes #30506 and Spans for Paths can be incorrect #39450 (PR macros: improve Span's expansion information #40597).
• Land macro behind a feature gate (PR Initial implementation of declarative macros 2.0 #40847).
• Future-proof matchers (e.g. $e:expr) by employing a simpler, more general grammar.
• Allow fragments (e.g. $e where $e:expr) to be parsed in more contexts (c.f. Macro matching is incorrect for macros invoked in macro expansions #26361).
• Decide whether we want macro invocations in identifier positions and/or eager expansion.
• Remove $:meta matcher (Remove $:meta matcher in decl macros #49629)

Potentially blocking issues:

• Declarative macros: no error about duplicate trait items #71614

[CryZe]

@nrc Typo in Issue Name: "issuse"

[jseyfried]

Tasks

(dtolnay edit: moved the checklist up to the OP)

[nrc]

We need to RFC a whole bunch of stuff here. In particular, I would like to propose some new syntax for declaring macros and we should RFC the changes to matchers.

[tikue]

Can the hygiene RFC mention pattern hygiene? This in particular scares me:

// Unsuspecting user's code
#[allow(non_camel_case_types)]
struct i(i64);

macro_rules! ignorant_macro {
    () => {
        let i = 0;
        println!("{}", i);
    };
}

fn main() {
    // oh no!
    ignorant_macro!();
}

[jseyfried]

@tikue I'm not sure patterns need special treatment with respect to hygiene.

For example, on the hygiene prototype,

#[allow(non_camel_case_types)]
struct i(i64);

macro ignorant_macro() {
    let i = 0; // ERROR: let bindings cannot shadow tuple structs
    println!("{}", i); 
}

fn main() {
    ignorant_macro!(); // NOTE: in this macro invocation
}

This makes sense to me since let i = 0; is shadowing a tuple struct.
In particular, if let i = 0; were removed then the following use of i would resolve to the tuple struct, no matter where ignorant_macro is used.

Note the symmetry to this example:

#[allow(non_camel_case_types)]
struct i(i64);

fn ignorant_fn() {
    let i = 0; // ERROR: let bindings cannot shadow tuple structs
    println!("{}", i); 
}

If the tuple struct i isn't in scope at macro ignorant_macro() { ... }, then the let i = 0; does not shadow it and there is no error. For example, the following compiles on the hygiene prototype:

mod foo {
    pub macro ignorant_macro() {
        let i = 0;
        println!("{}", i); // Without `let i = 0;`, there would be no `i` in scope here.
    }
}

// Unsuspecting user's code
#[allow(non_camel_case_types)]
struct i(i64);

fn main() {
    foo::ignorant_macro!();
}

[alexreg]

Has there been any progress on Macros 2.0 lately?

[mark-i-m]

Note for those who haven't seen yet: macros 2.0 is apparently slated to be stable later this year, according to the proposed roadmap (rust-lang/rfcs#2314)...

On the one hand, that's pretty exciting 🎉!

On the other hand, I was really surprised that the feature is so close to being done and so little is known about it by the broader community... The RFC is really vague. The unstable book only has a link to this issue. This issue(s) in the issue tracker mostly have detailed technical discussions. And I can't find that much info anywhere about what's changed or how stuff works.

I don't mean to complain, and I really truly appreciate all the hard work by those implementing, but I would also appreciate more transparency on this.

[alexreg]

@marcbowes Yeah, I'm kind of worried too. It seems like there's quite a lot of work left for stabilisation this year. I offered to work on opt-out hygiene for identifiers myself, but have received no response yet... Some greater transparency would be nice, as you say.

[petrochenkov]

Macros 2.0 are not even in the RFC stage yet - the @jseyfried's RFC linked in the issue was never submitted, there's only very high level RFC 1584.
The implementation is purely experimental and mostly unspecified, and large part of it (not related to hygiene) is reused from macro_rules! without fixing problems that macros 2.0 were supposed to fix.

Some greater transparency would be nice

The problem is that no work happen right now, so there's nothing to reveal :(
@nrc is busy, @jseyfried is busy, I worked on macros 2.0 a bit and would really like to dig into them more, but I'm busy too, unfortunately.
We need someone to adopt the feature, support and extend it, and gain expert-level knowledge of it, otherwise we will end up breaking hygiene and syntax details left and right after stabilization.

[alexreg]

@petrochenkov Ah, fair enough. I mean, that's a shame, but it makes sense at least. I guess this is an open call to anyone who might be able to take ownership of this feature, since no one comes to mind? I could still have a go at a little sub-feature, but I'm certainly in no position to take ownership of this.

[petrochenkov]

@alexreg

I could still have a go at a little sub-feature

Yes, please do!
Hygiene opt-out is one of the primary missing parts and implementing it would be useful in any case.

IIRC, two questions will need to be decided on during implementation:

• Syntax for "unhygienic" identifiers (@jseyfried tentatively suggested #ident in Initial implementation of declarative macros 2.0 #40847)
• What exactly hygienic context the identifier introduced with #ident in a macro m will have - context of m's invocation? context after expanding all macros ("no hygiene")? something else?

[alexreg]

Syntax for "unhygienic" identifiers (@jseyfried tentatively suggested #ident in #40847)

Yeah, this was the plan. :-)

What exactly hygienic context the identifier introduced with #ident in a macro m will have - context of m's invocation? context after expanding all macros ("no hygiene")? something else?

What's your inclination? I'm leaning towards the context of m's invocation, but curious to hear your thoughts...

[danii]

Should metavariables be prevented from being named self or as keywords? I feel like metavariable keywords could be used for future expansions to macros, like associated macros with $self.

[danielrab]

yeah, I agree, no keywords should probably be valid names for metavariables, in case we want to make any of them special somehow

[eap314]

These suggestions might be out of scope for this discussion, but it beats the RFC process, so

• hygiene could be (mostly) achieved by having macros (mangle) prefix functions with h_ and normal functions be (mangle) prefixed uh_
• If you don't want hygiene, you could use a keyword like unhygienic to prefix with uh_

unhygienic macro count {}
macro count2 {}

count!();
unhygienic { count2!(); }

// hygiene bending
macro count3 {
() => { unhygienic { let x = 4 } }
}

To bring up the out of scope part,

• Proc macros can be preprocessed, removing the need for a separate crate
• You can proc use a crate to bring it into scope during preprocessing

[A1-Triard]

* hygiene could be (mostly) achieved by having macros (mangle) prefix functions with h_ and normal functions be (mangle) 

As far as I understand it would not work for #[no_mangle] functions. If this is correct, then it means the suggestion just does not work.

[lygstate]

When this will be stablized :)

[clarfonthey]

Presumably it will be stabilised after all of the requisite tasks in the issue description are completed.

[cybersoulK]

I might not fully grasp the nuances of macros 2.0, but these are the benefits I anticipate:

1 - I should be able to use crates, types, and functions that I define in my library directly without needing $crate (suggesting deprecation of $crate).
2 - Every item created within the macro should expand; after all, that's the core purpose of macros.

This reasoning is sound:

If you desire certain types or functions to remain private, simply define them outside the macro where they remain accessible to the macro.
If an item is dynamic and changes based on token input, it still needs to be expanded into the scope. Given that it's a distinct type, it shouldn't just disappear.
The types or variables introduced during expansion shouldn't conflict with tokens, even if their names are identical. Think of library-defined code as being in red, and tokens in blue; they are distinct entities.
The user decides where the macro expands. They can enclose it using a mod { macro!() } or restrict its scope using a block { macro!() }.
At its core, macros operate on this principle: Library code + User code (Tokens) = Output Expansion.
I'm concerned that we might be overcomplicating the concept of hygiene.

[cybersoulK]

From my previous points, I believe the confusion we're facing stems from how to handle situations where Library code intertwines with User code.
It's naturally expected for them to merge within macros, since, by definition, a macro represents the library code's blueprint being replicated into the user's domain. Perhaps there's a need for enhanced tools to manage instances when two textual terms conflict, rather than sidestepping the issue and introducing myriad limitations to macros.

---

yea, i don't see why this is hard to do, if the macro has a chance of word collision.

mod macro_mod {
macro!();
}

pub use macro_mod::Type as NewNamedType;

---

i would not be against if the macro 2.0 used mod by default when expanding in open scopes, such as

use crate::macro;
macro!();
pub use macro::Type as NewNamedType;

use crate::macro as macro_renamed;
macro_renamed!();
pub use macro_renamed::*;

//automatically created macro mod and macro_renamed mod.

But then there must be a differentiation of macros that contain function calls and variable assignments, and should be expanded directly, such as:

match ... {
   A(name) => macro!(name),
}

[jjpe]

@cybersoulK do everyone here a favor please, and don't multipost. Everyone who is subscribed to this gets a separate email about each post, and the signal:noise ratio is dropping real fast ATM.

To be clear: posting is fine, but maybe don't drive people up the wall with constant useless notifications.

[dengyunsheng250]

Has there been any progress on Macros 2.0 lately?

[jhpratt]

No, but there will hopefully be a general increase in knowledge in what the issues are, as the proposal for a macros working group has been accepted. The WG will be created shortly!

[tgross35]

It could be interesting to allow a way to reverse parsing to match from right to left. This can be faked using an accumulator and recursively popping tokens until you get a match on a single one, but this is a pretty messy pattern. A more simple way to create a reverse TT muncher would make it much easier to apply precedence to contents, such as when evaluating math.

// Works
macro first {
    ($first:tt $($rest:tt)*) => { println!("first item is {}", $first) };
}

// Fails to compile (multiple parsing options)
macro last {
    #[reverse] // maybe this could make it work
    ($($rest:tt)* $last:tt) => { println!("last item is {}", $last) }
}

Edit: an alternative to this would just be adjusting the rules for multiple parsing such that it tries right to left if it gets stuck. I think this would accomplish the same thing without needing an annotation.

[clarfonthey]

The main issue with RTL parsing is that, at least for now, parsing doesn't actually store the tokens being returned in a way that lets you iterate backwards. You could allow this and do it without allocating if you permit parsing the tokens twice (once to find the end of the macro invocation, the second time to do the actual parsing), but it's still a performance penalty to allow parsing from the other direction.

Of course, the usual way, recursively expanding an intermediate macro in reverse order, is probably slower than offering a dedicated method that allows this. If this were done, it would be nice to also have proc_macro::TokenStream implement DoubleEndedIterator to reflect that, and allow proc macros to also take advantage of that.

[Ygg01]

Here is to hoping two big issues are getting solved (hygiene bending and nesting macros).

I tried solving macro hygiene but one pre-requisite was learning how other languages do it. Learning Racket is where I lost my motivation.

[petrochenkov]

Here is to hoping two big issues are getting solved (hygiene bending and nesting macros).

I tried solving macro hygiene but one pre-requisite was learning how other languages do it. Learning Racket is where I lost my motivation.

The issue about hygiene is "Tracking issue for Span::def_site()", this issue is blocked on it.
In general, declarative macros are a syntactic sugar for procedural macros, so the "procedural macros 2.0" need to be implemented first.

Racket needs to be learned just enough to understand https://users.cs.utah.edu/plt/scope-sets/ in detail, to be able to possibly tweak it to match Rust realities.

[lolbinarycat]

one usecase of macros where rust does worse than C is local unhyginic macros used in the definition of a single array or function.

some way to bind specific variables to to those from the outer scope, or to disable macro hygine for certain identifies, would be appreciated.

perhaps i need to call a function 10 times, and each time i need to pass 7 different arguments, 6 of which are always the same local variables, and only the 7th changes. C could easily manage this with a define and undef, but in rust, you would have to pass all the local variables to each macro call, which wouldn't actually simplify the code at all.

[felix91gr]

which wouldn't actually simplify the code at all.

That is true for this version of "simplify":

Simpler code: shorter code.

But it is not true for the following version of "simplify":

Simpler code: code that is easier to understand and maintain.

I don't think locally disabling hygiene helps at all, if what we seek is simpler code in the second sense.

[ifsheldon]

one usecase of macros where rust does worse than C is local unhyginic macros used in the definition of a single array or function.

I think "worse" here is generally better. I've read a pretty bad C codebase once, which is full of unhygine macros that means a lot of sloppy auto copy-and-paste. IDEs could not even help me to analyze the code. I really appreciate Rust didn't just choose the simple way to do macros.

perhaps i need to call a function 10 times, and each time i need to pass 7 different arguments, 6 of which are always the same local variables, and only the 7th changes.

Make a closure please.

[tmccombs]

Make a closure please.

A closure can't do everything a macro can. For example it can't return from the parent function.

[mayabyte]

perhaps i need to call a function 10 times, and each time i need to pass 7 different arguments, 6 of which are always the same local variables, and only the 7th changes.

I think what you're describing is partial application which is something that can already be achieved with existing crates. Combining this with the Try operator could achieve the early-return functionality described above as well.

[lolbinarycat]

it turns out the existing macro_rules! macros can already access identifiers from their parent scope, so this type of local macro actually already works, for when you need to do something that partial application can't do.

[oli-obk]

Tracking issues are not intended as places for discussions. This issue already has over 100 comments and lots of subscribers. Please open separate issues and link them, instead of having the discussion here.