quote_* macros lose hygiene, gensym and span information

[huonw]

The quote_* macros stringify everything, which loses vital information.

// gensym.rs
#![feature(plugin_registrar, managed_boxes, quote)]
#![crate_type = "dylib"]

extern crate syntax;
extern crate rustc;

use syntax::ast;
use syntax::codemap::{Span};
use syntax::ext::base;
use syntax::ext::base::{ExtCtxt, MacExpr};
use syntax::ext::build::AstBuilder;
use syntax::parse::token;
use rustc::plugin::Registry;

#[plugin_registrar]
pub fn plugin_registrar(reg: &mut Registry) {
    reg.register_macro("test_quote", expand_syntax_ext);
}

pub fn expand_syntax_ext(cx: &mut ExtCtxt, sp: Span, _: &[ast::TokenTree]) -> Box<base::MacResult> {
    // expand to `{ let foo = true; foo }`, with a gensym'd foo.
    let ident = token::gensym_ident("foo");
    let decl = quote_stmt!(&mut *cx, let $ident = true;);
    let result = cx.expr_block(cx.block(sp, vec![decl], Some(cx.expr_ident(sp, ident))));

    println!("{}", result);

    MacExpr::new(result)
}

// test_gensym.rs
#![feature(phase)]

#[phase(plugin)] extern crate gensym;

fn main() {
    let a = test_quote!();
}

[...]
test_gensym.rs:6:13: 6:27 error: unresolved name `foo`.
test_gensym.rs:6     let a = test_quote!();
                             ^~~~~~~~~~~~~~
test_gensym.rs:1:1: 7:1 note: in expansion of test_quote!
test_gensym.rs:6:13: 6:27 note: expansion site
error: aborting due to previous error

Beautified/trimmed version of the output of the println!

Expr {
    node: ExprBlock(Block {
        view_items: [],
        stmts: [Spanned {
            node: StmtDecl(Spanned {
                node: DeclLocal(Local {
                    ty: Ty {
                        node: TyInfer,
                        span: Span {
                            lo: BytePos(7677),
                            hi: BytePos(7677),
                            expn_info: None
                        }
                    },
                    pat: Pat {
                        node: PatIdent(BindByValue(MutImmutable), Spanned {
                            node: "foo" (269),
                            span: Span {
                                lo: BytePos(7677),
                                hi: BytePos(7680),
                                expn_info: None
                            }
                        }, None),
                        span: Span {
                            lo: BytePos(7677),
                            hi: BytePos(7680),
                            expn_info: None
                        }
                    },
                    init: Some(Expr {
                        node: ExprLit(Spanned {
                            node: LitBool(true),
                            span: Span {
                                lo: BytePos(83),
                                hi: BytePos(97),
                                expn_info: None
                            }
                        }),
                        span: Span {
                            lo: BytePos(83),
                            hi: BytePos(97),
                            expn_info: None
                        }
                    }),
                    span: Span {
                        lo: BytePos(7677),
                        hi: BytePos(97),
                        expn_info: None
                    },
                    source: LocalLet
                }),
            }, 4294967295),
        }],
        expr: Some(Expr {
            node: ExprPath(Path {
                global: false,
                segments: [PathSegment {
                    identifier: "foo" (268)# 0,
                    lifetimes: [],
                    types: OwnedSlice {
                        {}
                    }
                }]
            }),
        }),
        rules: DefaultBlock,
    }),
}

Of particular attention is the difference in Name between node: "foo" (269)
identifier: "foo" (268)# 0,, and also the 7677 etc. numbers in the spans inside theDeclLocal.

[huonw]

cc @jbclements

[huonw]

cc #15750

[huonw]

Maybe we could hack something for now by putting the information into the stringified source, e.g. an identifier Ident { name: 10, ctxt: 99 } could be stringified as \x0010,99\x00 (i.e. just use the name and ctxt, not the string, with some special leading byte), and have quoting put the parser into special mode which parses a \x00...\x00 as an identifier.

(Huge hack...)

[pnkfelix]

@huonw that's actually not a terrible idea, at least as a short term hack ... I don't see any reason why we'd be on the hook to support that interface long term, as long as we (as you say) have quote_expr! be the thing that turns on the special parsing mode.

[jbclements]

I agree with @pnkfelix ; the string representation has only ad-hoc structure, so this change is actually improving it somewhat. I would be cautious, though; I can easily imagine this leaking out into userland so that programmers start using this representation as a way of hacking hygiene. There's nothing wrong with hacking hygiene, of course, but this probably isn't the way you'd want to do it.

[pnkfelix]

I think I have the aforementioned hack implemented. I hope to have a PR soon.

[pnkfelix]

cc @larsbergstrom

[larsbergstrom]

Thanks, @pnkfelix!

cc @kmcallister

[pnkfelix]

@huonw it is possible that I was premature in closing this issue with my PR. In particular, the hack fixes the problem of hygiene on Idents, but ... I don't know about gensym ... and it certainly does not address the span issue, right?

(oh wait, my PR didn't close this issue, I just thought it did...)

[huonw]

Printing names should handle gensym info correctly. The span issue is less pressing, as it's "just" diagnostics (rather than making functionality impossible to express), and could be temporarily replaced by syntax providing a "respanning" Fold that just replaces the DUMMY_SP with some given span.

[kmcallister]

src/libsyntax/ext/quote.rs contains an interesting note about the span issue:

    // We also bind a single value, sp, to ext_cx.call_site()
    //
    // This causes every span in a token-tree quote to be attributed to the
    // call site of the extension using the quote. We can't really do much
    // better since the source of the quote may well be in a library that
    // was not even parsed by this compilation run, that the user has no
    // source code for (eg. in libsyntax, which they're just _using_).
    //
    // The old quasiquoter had an elaborate mechanism for denoting input
    // file locations from which quotes originated; unfortunately this
    // relied on feeding the source string of the quote back into the
    // compiler (which we don't really want to do) and, in any case, only
    // pushed the problem a very small step further back: an error
    // resulting from a parse of the resulting quote is still attributed to
    // the site the string literal occurred, which was in a source file
    // _other_ than the one the user has control over. For example, an
    // error in a quote from the protocol compiler, invoked in user code
    // using macro_rules! for example, will be attributed to the macro_rules.rs
    // file in libsyntax, which the user might not even have source to (unless
    // they happen to have a compiler on hand). Over all, the phase distinction
    // just makes quotes "hard to attribute". Possibly this could be fixed
    // by recreating some of the original qq machinery in the tt regime
    // (pushing fake FileMaps onto the parser to account for original sites
    // of quotes, for example) but at this point it seems not likely to be
    // worth the hassle.

[kmcallister]

I believe all of this can be fixed in a library post-1.0, albeit not as cleanly.