pass macro call location to macro-function (rebase)

[vtjnash]

rebase of #13339
fixes #9577
replaces #9579
replaces #20895

• merge and tag Compat
• merge and tag Documenter
• re-enable building docs
• add to NEWs and docs on macros

[Keno]

I can haz column info as well?

[vtjnash]

The parser doesn't have any, afaik. I think getting line number ranges would be the next step.

[Keno]

#9579 has code for column tracking in the parser. I don't really care about line ranges since that's easy to compute for string macros.

[vtjnash]

Would you recommend calling this __source__ or __location__ instead then, and update the LineNumberNode object to be a general LocationNode (with possible future fields line, column, file, line_end, and column_end)?

[Keno]

That sounds like a good idea.

[vtjnash]

This breaks Documenter because it tries to pattern-match the macrocall expression to see if it has zero-arguments or no arguments specified, and will need the following patch:

diff --git a/src/DocSystem.jl b/src/DocSystem.jl
index e3b89c3f..333feabe 100644
--- a/src/DocSystem.jl
+++ b/src/DocSystem.jl
@@ -88,7 +88,7 @@ binding(m::Module, λ::Any) = binding(λ)
 
 function signature(x, str::AbstractString)
     ts = Base.Docs.signature(x)
-    (Meta.isexpr(x, :macrocall, 1) && !endswith(strip(str), "()")) ? :(Union{}) : ts
+    (Meta.isexpr(x, :macrocall, 2) && !endswith(strip(str), "()")) ? :(Union{}) : ts
 end
 if VERSION < v"0.5.0-dev"
     Base.Docs.signature(::Any) = :(Union{})
diff --git a/src/Utilities/Utilities.jl b/src/Utilities/Utilities.jl
index c4ce0a28..6c477443 100644
--- a/src/Utilities/Utilities.jl
+++ b/src/Utilities/Utilities.jl
@@ -238,7 +238,7 @@ Returns a expression that, when evaluated, returns an [`Object`](@ref) represent
 function object(ex::Union{Symbol, Expr}, str::AbstractString)
     binding   = Expr(:call, Binding, splitexpr(Docs.namify(ex))...)
     signature = Base.Docs.signature(ex)
-    isexpr(ex, :macrocall, 1) && !endswith(str, "()") && (signature = :(Union{}))
+    isexpr(ex, :macrocall, 2) && !endswith(str, "()") && (signature = :(Union{}))
     Expr(:call, Object, binding, signature)
 end
 
@@ -275,7 +275,7 @@ if VERSION < v"0.5-"
     end
 else
     function docs(ex::Union{Symbol, Expr}, str::AbstractString)
-        isexpr(ex, :macrocall, 1) && !endswith(rstrip(str), "()") && (ex = quot(ex))
+        isexpr(ex, :macrocall, 2) && !endswith(rstrip(str), "()") && (ex = quot(ex))
         :(Base.Docs.@doc $ex)
     end
 end

[tkelman]

Does this need devdocs updates for AST changes?

[tkelman]

needs to be added to stdlib doc index to be included in the rendered docs

[tkelman]

end sentence with a period not a comma

[vtjnash]

For users needing compatible parsing, note that (@__LINE__) will work.

[tkelman]

duplicated "the"

[StefanKarpinski]

#9577, #13339, #21746

[c42f]

Ah, it's suddenly clear how this approach could work at all. Very clever!

[c42f]

@vtjnash @ihnorton with this approach, it looks like there's no way to create a wrapper macro which should be transparent to @LINE, in the same way that non-block quote doesn't generate normal line number nodes.

In MicroLogging I've got @info, @warn implemented as trivial wrappers around an @logmsg macro which has all the guts of the logging implementation, and I was hoping to have @LINE expanded inside @logmsg. Of course, for the particular purposes of my package I can work around this limitation, but I'd rather not have to.

Do you see a viable way to allow such wrapper macros using the __source__ approach you've implemented here?

One probably ugly possibility would be to have access to a Expr(:macrocall_with_source, ...) (better name needed!) where the user can pass the __source__ argument explicitly, bypassing the expansion step?

[vtjnash]

From inside one macro, I think both
return Expr(:macrocall, Symbol("@macroname"), __source__, args...)
or equivalently
m = :(@macroname args...); m.args[1] = __source__; return m
should work.

[c42f]

Ah, of course. Thanks - this does make a lot of sense. (The reason I ask is that I'm having odd LLVM compile errors trying to build julia-0.7 right now, so I haven't been able to just try the code myself.)

This seems to be both the benefit and downside of the solution here: a benefit, because the __source__ should be clearly visible and non-magical in the AST. This is great. A downside, because it'll disturb any code which processes Expr(:macrocall, ...) (which quite honestly probably isn't a whole lot of user code).

[tkelman]

The Expr(:macrocall, ...) change broke multiple packages on nightly. A deprecation should really be put in.

[c42f]

Hah, wouldn't have been a problem with #20895 :-P

How could a deprecation be made to work? I assume this broke packages which were doing moderately "interesting" AST manipulations. Those would just be manipulating the surface syntax via an Expr which happened to have head==:macrocall so there's nowhere to insert any kind of depwarn, is there?

I could imagine adding some kind of AST pattern matching tool inside Compat, and encouraging people to use that when interacting with ASTs containing macros? Could be a lot easier on users than the minimal flag in https://github.com/JuliaLang/Compat.jl/pull/355/files

[tkelman]

there've been a few similar deprecations put in at the lowering level for macro writers who construct obsolete AST forms, instead of hard-breaking them

[c42f]

Ok, so you're referring to packages which construct an Expr(:macrocall, args...) explicitly? In that case, would it be sufficient to detect when the first argument isn't a line number node, and add an extra dummy __source__ argument just before calling invoke-julia-macro?

There's another case which could cause breakage, as @a 1 now parses to include the value of the __source__ explicitly in the AST. Packages which attempt to process this AST into something else are presumably going to be surprised, when the Expr ends up with 1 in the args[3] slot rather than args[2]?