Llogiq on stuff

proc_macro_attribute Revisited

Recently, the procedural macro interface was somewhat stabilized. OK, there’s still the unstable proc_macro_hygiene feature you have to activate, but at least the registrar and rustc_private are no longer needed.

So I set out to rewrite flamer, my most simple procedural macro crate, to the new interface, using syn and quote. Given that this new interface is deemed to stabilize in the next months, it’s as good a time as ever to take the plunge.

So, first, the Cargo.toml. We no longer need type = plugin. Instead, our [lib] gets a proc_macro = true setting. We also need some dependencies (current as of the time of writing, please check the versions if you intend to copy):

syn = { version = "0.15.18", features = ["full", "fold", "parsing"] }
quote = "0.6.9"

Note the used syn features. full means syn comes with the complete functionality to parse and fold any Rust AST type, whereas by default, only types needed for derives are included (which greatly reduces compile times). The fold feature activates the Fold trait and corresponding functions giving us a nice interface to change code on the fly. The parsing feature allows parsing TokenStreams into syn’s types.

Compared to my initial version, we no longer need to deal with registering anything. We just annotate a plain function with an attribute and voilà:

#[proc_macro_attribute]
fn flame(attrs: TokenStream, code: TokenStream) -> TokenStream {
    ..
}

The attrs stream denotes the given attributes (one of which is our flame attribute). The function name is the name of the attribute that activates the macro.

syn has a parse_macro_input! macro that we can use to parse TokenStreams into anything that has a Parse implementation. It’s not too hard to create one ourselves, and indeed this is one thing I do for flamer to allow setting an initial name. In this case however, we want to parse the code into an Item so we can fold it:

let input = parse_macro_input!(code as Item);
let mut flamer = parse_macro_input!(attrs as Flamer);
let item = fold::fold_item(&mut flamer, input);
TokenStream::from(quote!(#item))

The Fold trait is pretty straightforward, with one method per type that can be folded and that are by default implemented by folding all the contents. In our case, we want to be able to special-case arbitrary items to ensure that double-#[flame]s don’t get counted twice and #[noflame]d items don’t get flamed at all. We also record the names of all items visited to generate useful names for our flame guards later.

We also want to insert a let _flame_guard = ::flame::start_guard(#name) statement (where name is the string literal generated by joining the recorded item names). at the starting position of the outermost block of each #[flame]d function or method. The parse_quote!(..) macro makes generating such a statement straightforward and as we already get the whole object, we can mutate it by block.stmts.insert(0, ..). This works without any surprise. Great!

I won’t list any code here, but you can simply look into the flamer github repository, the code is quite simple.

Next, I wanted to rewrite overflower. This is naturally a bit more complex, as we need to change ExprUnary and ExprBinarys into ExprCalls and obey attributes on various items and expressions. One niggle here is that syn keeps the attributes in the inner types, so Expr is a big enum where each variant has its own content type, e.g. ExprBinary. This means I have to match over all variants that could contain relevant subexpressions to take care of the attributes. An accessor on Expr would have been very helpful here.

When I finally thought that I had it figured out, I got an unhelpful error during macro expanson. cargo expand only spat out the same error, which didn’t help either, so I commented out various parts of the code to see whether they’d make any difference. This at least reduced the possible errors to the generation of the trait method calls. The problem is that a &str cannot be quoted as an Ident.

I finally arrived at the solution: syn::parse_str::<Ident>(..).unwrap(). This leaves me with one final problem: I cannot get macros expanded. This used to be a problem with the old interface, too, but there were a number of workarounds.

In fact, for both overflower and mutagen, I don’t even need expanded macros, if I can get the macro matchers for their arguments. For example, if I have the following macro call: id!(a + b), I cannot know if a + b is an Expr or if a and b are paths and the + is matched by the macro. So I risk changing the macro’s meaning if I blindly try to fold expressions within their argument TokenStreams, and in the worst case failing the build.

It’ll be very interesting to see the solutions to that problem. In the meantime anyone working on this, chat me up on twitter, reddit or the next RustFest!