Make proc_macro available outside procedural macro generation

[dead-claudia]

Proposal

Problem statement

proc_macro has no clear testing story on its own. It's also not possible to use it in build scripts.

Motivating examples or use cases

The top three most downloaded crates are Syn, proc-macro2, and quote.

• proc-macro2 is mostly just a port of the built-in proc_macro crate.
• The functionality of quote already has an unstable analogue in proc_macro itself.
• Syn parses the token stream into an AST for further manipulation. (Note: integrating this in any way into proc_macro is explicitly outside the scope of this proposal.)

The README introduction for proc-macro2 could itself just be this entire section:

A wrapper around the procedural macro API of the compiler's proc_macro crate.
This library serves two purposes:

• Bring proc-macro-like functionality to other contexts like build.rs and
  main.rs. Types from proc_macro are entirely specific to procedural macros
  and cannot ever exist in code outside of a procedural macro. Meanwhile
  proc_macro2 types may exist anywhere including non-macro code. By developing
  foundational libraries like [syn] and [quote] against proc_macro2 rather
  than proc_macro, the procedural macro ecosystem becomes easily applicable to
  many other use cases and we avoid reimplementing non-macro equivalents of
  those libraries.

• Make procedural macros unit testable. As a consequence of being specific
  to procedural macros, nothing that uses proc_macro can be executed from a
  unit test. In order for helper libraries or components of a macro to be
  testable in isolation, they must be implemented using proc_macro2.

Solution sketch

Expose the built-in proc_macro crate outside procedural macros, similar to what's currently being done (in nightly) for test. Not sure how exactly that'd work on the technical side, but the external design concept is at least simple. 🙂

Alternatives

Obviously, it's already being done as a crate. This work is highly duplicative, however, and makes little sense to have two separate projects maintain two separate pieces of functionality that do basically the same thing. It makes even less sense considering you all sometimes have to proactively work with their maintainers and around their crates just to avoid breaking people.

Links and related work

See the section "Motivating examples or use cases". It applies here as well.

What happens now?

This issue contains an API change proposal (or ACP) and is part of the libs-api team feature lifecycle. Once this issue is filed, the libs-api team will review open proposals as capability becomes available. Current response times do not have a clear estimate, but may be up to several months.

Possible responses

The libs team may respond in various different ways. First, the team will consider the problem (this doesn't require any concrete solution or alternatives to have been proposed):

• We think this problem seems worth solving, and the standard library might be the right place to solve it.
• We think that this probably doesn't belong in the standard library.

Second, if there's a concrete solution:

• We think this specific solution looks roughly right, approved, you or someone else should implement this. (Further review will still happen on the subsequent implementation PR.)
• We're not sure this is the right solution, and the alternatives or other materials don't give us enough information to be sure about that. Here are some questions we have that aren't answered, or rough ideas about alternatives we'd want to see discussed.

[joshtriplett]

We discussed this in today's @rust-lang/libs-api meeting.

We were generally favorable towards taking this step to make proc_macro work everywhere.

In order to approve this, we'd need to see a few things:

• A review of the current proc_macro API, with specific proposal for how the API behaves in a non-proc-macro context. For instance, should the diagnostics APIs panic, or return an error?

• A proposed implementation strategy. proc_macro currently uses a u32 handle to a rustc object. proc_macro2 uses the approach of an enum containing either a handle or a full data structure, and those data structures are larger, which would make it larger for everyone. Another approach would be to always use a u32 handle, and in the non-proc-macro case have that be an index in an internal array; that would avoid making the data structures any larger.

[dead-claudia]

• A proposed implementation strategy. proc_macro currently uses a u32 handle to a rustc object. proc_macro2 uses the approach of an enum containing either a handle or a full data structure, and those data structures are larger, which would make it larger for everyone. Another approach would be to always use a u32 handle, and in the non-proc-macro case have that be an index in an internal array; that would avoid making the data structures any larger.

@joshtriplett Have you all considered a third alternative of just #[cfg]ing the layout based on whether it's truly operating as a macro or not? That might be able to sidestep the issue entirely, and since you're the compiler, you'd also be able to just apply a cfg feature flag to the crate if needed.

[dtolnay]

@dead-claudia yes this was discussed in the same meeting. That would involve increasing the size of the build graph, which is implementable but is a pretty severe tradeoff that to me seems unlikely to be the best way if we can get any other way to work.

For example consider having a build-dependency and a proc macro dependency with some overlap between their transitive dependencies. Today the overlap would be compiled once — even in Cargo's cross-compilation mode (--target) which sometimes rebuilds crates separately for the host and target. However in the implementation strategy you bring up it would need to compile every crate in that overlap twice, just in case libproc_macro is used anywhere in them.

[joshtriplett]

@dead-claudia Please ping us when you've updated the ACP, and we'll be happy to review.

[joshtriplett]

Following up on this. ping @dead-claudia

[dead-claudia]

Sorry, this is not high-priority for me at the moment, so it may be a while before I can come up with something. (Currently, my focus is back in the JS world.)

Edit: it's still on my radar, to be clear. Just residing in the back of my mind. Also, I'm not really familiar with how the compiler is structured at all beyond just some high-level algorithmic details, so pardon my ignorance on that.

[joshtriplett]

We'd be quite interested in seeing this work happen. It might make sense to turn this into a project goal. (Would you be interested in owning that or should we seek another owner?)

[dead-claudia]

@joshtriplett If you want this to be resolved with any sort of speed, best to pick another owner, unfortunately.

[Amanieu]

We discussed this in the libs-api meeting and are accepting this in principle even though the design isn't fully sketched out. Further design work should happen on the tracking issue.