Decouple proc_macro from the rest of the compiler. #49219

eddyb · 2018-03-20T21:56:23Z

This PR removes all dependencies of proc_macro on compiler crates and allows multiple copies of proc_macro, built even by different compilers (but from the same source), to interoperate.

Practically, it allows:

running proc macro tests at stage1 (I moved most from -fulldeps to the regular suites)
using proc macros in the compiler itself (may require some rustbuild trickery)

On the server (i.e. compiler front-end) side:

server::* traits are implemented to provide the concrete types and methods
- the concrete types are completely separated from the proc_macro public API
- the only use of the type implementing Server is to be passed to Client::run

On the client (i.e. proc macro) side (potentially using a different proc_macro instance!):

client::Client wraps around client-side (expansion) function pointers
- it encapsulates the proc_macro instance used by the client
- its run method can be called by a server, to execute the client-side function
  - the client instance is bridged to the provided server, while it runs
  - ~~currently a thread is spawned, could use process isolation in the future~~
    (not the case anymore, see Run proc macro invocations in separate threads. #56058)
proc macro crates get a generated static holding a &[ProcMacro]
- this describes all derives/attr/bang proc macros, replacing the "registrar" function
- each variant of ProcMacro contains an appropriately typed Client<fn(...) -> ...>

proc_macro public APIs call into the server via an internal "bridge":

only a currently running proc macro Client can interact with those APIs
- server code might not be able to (if it uses a different proc_macro instance)
  - however, it can always create and run its own Client, but that may be inefficient
the bridge uses serialization, C ABI and integer handles to avoid Rust ABI instability
each invocation of a proc macro results in disjoint integers in its proc_macro handles
- this prevents using values of those types across invocations (if they even can be kept)

r? @alexcrichton cc @jseyfried @nikomatsakis @Zoxc @thepowersgang

eddyb · 2018-03-20T21:56:36Z

@bors try

bors · 2018-03-20T21:56:41Z

🔒 Merge conflict

eddyb · 2018-03-20T22:01:25Z

@bors try

bors · 2018-03-20T22:01:35Z

⌛ Trying commit 6e63e8d with merge c4a77e1...

@alexcrichton

Decouple proc_macro from the rest of the compiler. This PR removes all dependencies of `proc_macro` on compiler crates and allows multiple copies of `proc_macro`, built even by different compilers (but from the same source), to interoperate. On the compiler (aka "frontend") side: * `Registry` is implemented (as until now) * instead of function pointers, the `Expand{1,2}` wrapper types are received * `FrontendInterface` is implemented to provide the concrete type and methods * the concrete types are completely separated from the `proc_macro` public API * the only use of the implementer is to be passed to `Expand{1,2}::run` On the proc macro side (potentially using a different `proc_macro` instance): * `&mut Registry` is passed to the registrar (as until now) * `Expand{1,2}` wrappers are created to be passed to the `Registry` * they encapsulate the `proc_macro` instance used by the macro crate * their `run` method will set up the "current frontend" of that instance `proc_macro` public APIs call into the "current frontend" via the internal `bridge`: * only a currently running proc macro can interact with those APIs * the frontend itself might not be able to (if it uses a different `proc_macro` instance) * the `bridge` uses C ABI to avoid Rust ABI instability and "generation tagging" for safety * each invocation of a proc macro results in a different "generation" * specifically, opaque `proc_macro` types wrapping concrete frontend types are tagged * this prevents using values of those types across invocations (even if they can be kept) * an ABI mismatch between two versions of `proc_macro` is only possible during stage1 * specifically, rustc compiled by stage0 but proc macros compiled by stage1 * can only affect running tests at stage1 or the compiler using proc macros * defficiencies in the `bridge` can be addressed without waiting for a release cycle r? @alexcrichton cc @jseyfried @nikomatsakis @Zoxc @thepowersgang

eddyb · 2018-03-20T22:12:12Z

cc @rust-lang/infra We shouldn't merge this without a crater run, although you might want to wait on a review as well (then again, the results could inform the review, so whichever comes first).

bors · 2018-03-21T00:23:15Z

☀️ Test successful - status-travis
State: approved= try=True

Zoxc · 2018-03-21T01:49:05Z

src/libproc_macro/lib.rs

+
+    #[unstable(feature = "proc_macro_internals", issue = "27812")]
+    #[doc(hidden)]
+    pub contents: Term,


Do we want to commit to using interning for literals? I'm not sure interning it terrible useful for that.

Wait, I can make these fields private again. Definitely not considering stabilizing any of this.

Zoxc · 2018-03-21T02:39:54Z

src/libtest/Cargo.toml

@@ -11,3 +11,6 @@ crate-type = ["dylib", "rlib"]
 [dependencies]
 getopts = "0.2"
 term = { path = "../libterm" }
+
+# not actually used but needed to always have proc_macro in the sysroot
+proc_macro = { path = "../libproc_macro" }


This should probably be a rustbuild change.

bjorn3 · 2018-03-21T17:26:46Z

Could you use something like bincode for data bridging, instead of 800 lines of own code?

eddyb · 2018-03-22T01:09:05Z

@bjorn3 I'm not serializing the data (but rather using handles), and bincode (like any other serialization) is annoying to use without derived impls (which you can't have because this is proc_macro itself - or at least I don't want to try to make that work, for now).

I am going to look into simplifying the bridge code, removing most unsafe, and making it more amenable to message-passing, however.

bjorn3 · 2018-03-22T07:11:16Z

I get it.

Mark-Simulacrum · 2018-03-22T22:38:34Z

Not sure if you wanted perf, or whether this is even helpful, but http://perf.rust-lang.org/compare.html?start=75af15ee6ca0c12b699a17984b033363cd25e9c3&end=c4a77e121949c30306dbbf48259a49c3aeb55028&stat=instructions%3Au

eddyb · 2018-03-23T13:39:34Z

@Mark-Simulacrum It doesn't look bad, that's for sure, thanks! If we switch to spawning a thread (or even a process) per invocation it'd warrant another perf run.

A few tests related to proc-macros fail when performing stage-1 testing. This PR adds // ignore-stage1 to them, along with a FIXME. Original issue: rust-lang#49352 This should (hopefully) be fixed when rust-lang#49219 is merged.

aidanhs · 2018-03-26T12:13:50Z

Crater run started

aidanhs · 2018-03-31T23:31:22Z

Hi @eddyb (crater requester), @Zoxc (PR reviewer)! Crater results are at: http://cargobomb-reports.s3.amazonaws.com/pr-49219/index.html. 'Blacklisted' crates (spurious failures etc) can be found here. If you see any spurious failures not on the list, please make a PR against that file.

(interested observers: Crater is a tool for testing the impact of changes on the crates.io ecosystem. You can find out more at the repo if you're curious)

petrochenkov · 2018-11-30T11:00:41Z

I can't believe you've done this!

This is only necessary until rust-lang/rust#49219 hits stable.

1.32 includes rust-lang/rust#49219, which means new cbindgen no longer depends on compiler internals, which fixes some reported build issues on IRC.

1.32 includes rust-lang/rust#49219, which means new cbindgen no longer depends on compiler internals, which fixes some reported build issues on IRC. UltraBlame original commit: 8e48fdd65475c48fdc2f6af9e359c3d07bee66e5

…atsakis Remove some `ignore-stage1` annotations. These tests appear to no longer need the `ignore-stage1` marker. - `run-make-fulldeps/issue-37839` and `run-make-fulldeps/issue-37893`: I believe these were due to the use of proc-macros, and probably were just missed in rust-lang#49219 which fixed the proc-macro compatibility. - `compile-fail/asm-src-loc-codegen-units.rs`: This was due to an old issue with landing pads (as mentioned in the linked issue rust-lang#20184). `-Zno-landing-pads` was an option when building the first stage (it was much faster), but somewhere along the way (I think the switch from makefiles to rustbuild), the option was removed. - NOTE: This test doesn't actually test what it was originally written for, and is probably mostly pointless now. This test was asserting the message "build without -C codegen-units for more exact errors", but that was removed in rust-lang#42682. It is now in essence identical to `asm-src-loc.rs`.

rust-highfive assigned alexcrichton Mar 20, 2018

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Mar 20, 2018

eddyb force-pushed the proc-macro-decouple branch from 67f317e to 6e63e8d Compare March 20, 2018 21:59

eddyb added the S-waiting-on-crater Status: Waiting on a crater run to be completed. label Mar 20, 2018

Zoxc reviewed Mar 21, 2018

View reviewed changes

This comment has been minimized.

Sign in to view

eddyb changed the title ~~Decouple proc_macro from the rest of the compiler.~~ [WIP] Decouple proc_macro from the rest of the compiler. Mar 22, 2018

eddyb force-pushed the proc-macro-decouple branch from 6e63e8d to cd9ac21 Compare March 22, 2018 20:07

This comment has been minimized.

Sign in to view

eddyb force-pushed the proc-macro-decouple branch from cd9ac21 to 76590d8 Compare March 23, 2018 19:13

hanna-kruppe mentioned this pull request Mar 25, 2018

Stage 1 test failure on master branch #49352

Closed

Phlosioneer mentioned this pull request Mar 25, 2018

Ignore proc macro stage 1 tests #49366

Closed

shepmaster removed the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Mar 30, 2018

eddyb mentioned this pull request Mar 30, 2018

Reimplement public interface for stability dtolnay/proc-macro2#71

Merged

eddyb deleted the proc-macro-decouple branch November 30, 2018 11:13

konstin mentioned this pull request Dec 1, 2018

Find a way to update serde_derive mozilla/cbindgen#203

Closed

petrochenkov mentioned this pull request Dec 8, 2018

Panic iterating over proc_macro::TokenStream #55640

Closed

hcpl mentioned this pull request Dec 9, 2018

how to create an ident from a string dtolnay/syn#546

Closed

sfackler added a commit to sfackler/proc-macro2 that referenced this pull request Dec 18, 2018

Don't link to proc_macro on musl

9e22ce2

This is only necessary until rust-lang/rust#49219 hits stable.

petrochenkov mentioned this pull request Dec 18, 2018

Remove TokenStream::JointTree. #56964

Merged

dtolnay mentioned this pull request Jan 2, 2019

Quote is hard to use for non-procmacro code generation dtolnay/quote#90

Closed

eddyb mentioned this pull request Mar 14, 2019

RFC: Stabilize the alloc crate rust-lang/rfcs#2480

Merged

nikomatsakis mentioned this pull request Mar 25, 2019

Query Parallelization Tracking Issue #48685

Open

31 tasks

Xanewok mentioned this pull request Apr 17, 2019

Use crates.io version of rustc-serialize #40527

Closed

ehuss mentioned this pull request Jul 3, 2020

Remove some ignore-stage1 annotations. #73981

Merged

mystor mentioned this pull request Jul 2, 2021

Optimize and improve the proc_macro RPC interface for cross-thread execution #86816

Closed

vlad20012 mentioned this pull request Jan 28, 2022

Add a way to disable dll copying for users of proc_macro_srv library rust-lang/rust-analyzer#11365

Merged

dtolnay mentioned this pull request May 25, 2023

Extern crate proc_macro available on all wasm targets dtolnay/proc-macro2#388

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Decouple proc_macro from the rest of the compiler. #49219

Decouple proc_macro from the rest of the compiler. #49219

eddyb commented Mar 20, 2018 •

edited

Loading

eddyb commented Mar 20, 2018

bors commented Mar 20, 2018

eddyb commented Mar 20, 2018

bors commented Mar 20, 2018

eddyb commented Mar 20, 2018

bors commented Mar 21, 2018

Zoxc Mar 21, 2018

eddyb Mar 21, 2018

Zoxc Mar 21, 2018

bjorn3 commented Mar 21, 2018

eddyb commented Mar 22, 2018

bjorn3 commented Mar 22, 2018

This comment has been minimized.

This comment has been minimized.

Mark-Simulacrum commented Mar 22, 2018

eddyb commented Mar 23, 2018

aidanhs commented Mar 26, 2018

aidanhs commented Mar 31, 2018

petrochenkov commented Nov 30, 2018

Decouple proc_macro from the rest of the compiler. #49219

Decouple proc_macro from the rest of the compiler. #49219

Conversation

eddyb commented Mar 20, 2018 • edited Loading

eddyb commented Mar 20, 2018

bors commented Mar 20, 2018

eddyb commented Mar 20, 2018

bors commented Mar 20, 2018

eddyb commented Mar 20, 2018

bors commented Mar 21, 2018

Zoxc Mar 21, 2018

Choose a reason for hiding this comment

eddyb Mar 21, 2018

Choose a reason for hiding this comment

Zoxc Mar 21, 2018

Choose a reason for hiding this comment

bjorn3 commented Mar 21, 2018

eddyb commented Mar 22, 2018

bjorn3 commented Mar 22, 2018

This comment has been minimized.

This comment has been minimized.

Mark-Simulacrum commented Mar 22, 2018

eddyb commented Mar 23, 2018

aidanhs commented Mar 26, 2018

aidanhs commented Mar 31, 2018

petrochenkov commented Nov 30, 2018

eddyb commented Mar 20, 2018 •

edited

Loading