TokenStream::parse does not resolve with Span::call_site()

[golddranks]

Updated description

The following procedural macro:

#![feature(proc_macro)]

extern crate proc_macro;

use proc_macro::*;

#[proc_macro]
pub fn foo(input: TokenStream) -> TokenStream {
    let mut body = Vec::<TokenTree>::new();

    // input = `let a = 2;`
    body.extend(input.into_iter());

    // manually add `let b = a;`
    body.push(Term::new("let", Span::call_site()).into());
    body.push(Term::new("b", Span::call_site()).into());
    body.push(Op::new('=', Spacing::Alone).into());
    body.push(Term::new("a", Span::call_site()).into());
    body.push(Op::new(';', Spacing::Alone).into());

    // parse and add `let c = b;`
    body.extend("let c = b;".parse::<TokenStream>().unwrap().into_iter());

    let mut ret = Vec::<TokenTree>::new();
    ret.push(Term::new("fn", Span::call_site()).into());
    ret.push(Term::new("main", Span::call_site()).into());
    ret.push(Group::new(Delimiter::Parenthesis, TokenStream::empty()).into());
    ret.push(Group::new(Delimiter::Brace, body.into_iter().collect()).into());
    ret.into_iter().collect()
}

fails when invoked as:

#![feature(proc_macro)]

extern crate foo;

foo::foo! {
    let a = 3;
}

with the error message:

error[E0425]: cannot find value `b` in this scope
 --> src/main.rs:6:1
  |
6 | / foo::foo! {
7 | |     let a = 3;
8 | | }
  | |_^ not found in this scope

error: aborting due to previous error

For more information about this error, try `rustc --explain E0425`.

Original description

Linking the futures-await crate and a crate that useserror-chain-derive together breaks the build. Here's a simple example: https://github.com/golddranks/test_futures_await_error_chain_derive Either of the two dependencies (futures-await and dotenv, which uses error-chain-derive) builds on its own (using the latest nightly), but together they refuse to build.

This seems to have something to do with the defsite hygiene: error-chain-derive is using derive macros 1.1, and the syntax for using it is as follows: (from https://github.com/purpliminal/rust-dotenv/blob/master/src/errors.rs )

#[derive(Debug, ErrorChain)]
#[cfg_attr(not(feature = "backtrace"), error_chain(backtrace = "false"))]
pub enum ErrorKind {
    // generic error string, required by derive_error_chain
    Msg(String),
    #[error_chain(custom)]
    #[error_chain(description = r#"|_| "Parsing Error""#)]
    #[error_chain(display = r#"|l| write!(f, "Error parsing line: '{}'", l)"#)]
    LineParse(String),
    #[error_chain(foreign)]
    Io(::std::io::Error),
    #[error_chain(foreign)]
    EnvVar(::std::env::VarError),
}

|l| write!(f, "Error parsing line: '{}'", l) breaks, because it can't find f: cannot find value f in this scope

It seems like just the presence of futures-await somehow affects how the Derive macros 1.1 hygiene works; however, the derive macros are stable, so this is an undesirable thing.

[golddranks]

Looking at the symptoms, likely to be the same bug as #50061

[golddranks]

This still happens on the latest nightly (79252ff4e 2018-04-29), even after the patches #50152 and alexcrichton@e934873 landed.

[golddranks]

Ping @petrochenkov @alexcrichton

[alexcrichton]

I've updated the OP and issue title to reflect the underlying issue, minimized at https://gist.github.com/alexcrichton/6b4c56549c6732e697eb0aa9986bf398.

The tl;dr; is that a TokenStream created through TokenStream::from_str (aka parsing code) cannot refer to tokens with the span Span::call_site() (nor tokens input to the macro itself). This means that for whatever reason the hygienic context applied to tokens created through parsing strings is different than that of Span::call_site, exposing "oddness" of hygiene.

@petrochenkov or @nrc, do y'all know where this might be occurring?

[petrochenkov]

@alexcrichton
A guess: this may be a regression from changes in src/libsyntax/ext/quote.rs in #49154 (DUMMY_SP -> ident.span for ast::Ident and ast::Lifetime).
Could you test whether reverting changes in those two lines fixes the regression?
If it doesn't, then I'll try to investigate further sometime during this week.

[alexcrichton]

Sure yeah, I will test out that hypothesis tomorrow morning!

[alexcrichton]

Ah unfortunately this diff on 5a662bf produced the same error:

$ ./build/x86_64-unknown-linux-gnu/stage2/bin/rustc foo.rs --crate-type proc-macro
$ ./build/x86_64-unknown-linux-gnu/stage2/bin/rustc bar.rs -L .
error[E0425]: cannot find value `b` in this scope
 --> bar.rs:5:1
  |
5 | / foo::foo! {
6 | |     let a = 3;
7 | | }
  | |_^ not found in this scope

error: aborting due to previous error

For more information about this error, try `rustc --explain E0425`.

[jan-hudec]

@alexcrichton,

The tl;dr; is that a TokenStream created through TokenStream::from_str (aka parsing code) cannot refer to tokens with the span Span::call_site() (nor tokens input to the macro itself). This means that for whatever reason the hygienic context applied to tokens created through parsing strings is different than that of Span::call_site, exposing "oddness" of hygiene.

I am probably missing something, but to me it sounds about right. I would expect manually parsed code to be its own context. That's the hygiene. Similar to literal code in template macros, except proc macros don't have definition site, so the context is simply fresh. To refer to the symbol from the call site, you can replace it with the Token from input or manually created with Span::call_site(), no?

[alexcrichton]

@jan-hudec I think you're right yeah, but the problem is that we've been advocating Macros 1.1 and 1.2 as "hygiene comes later, we're just copy/pasting code". That seems to not be the case and there's actual hygiene involved, which is unintended currently