Introducing Julia/Metaprogramming

What is metaprogramming?
Meta-programming is when you write Julia code to process and modify Julia code. With the meta-programming tools, you can write Julia code that modifies other parts of your source files, and even control if and when the modified code runs.

In Julia, the execution of raw source code takes place in two stages. (In reality there are more stages than this, but at this point we'll focus on just these two.)

Stage 1 is when your raw Julia code is parsed — converted into a form that is suitable for evaluation. You'll be familiar with this phase, because this is when all your syntax mistakes are noticed... The result of this is an abstract syntax tree or AST (Abstract Syntax Tree), a structure that contains all your code, but in a format that is easier to manipulate than the human-friendly syntax normally used.

Stage 2 is when that parsed code is executed. Usually, when you type code into the REPL and press Return, or when you run a Julia file from the command line, you don't notice the two stages, because they happen so quickly. However, with Julia's metaprogramming facilities, you can access the code after it's been parsed but before it's evaluated.

This lets you do things that you can't normally do. For example, you can convert simple expressions to more complicated expressions, or examine code before it runs and change it so that it runs faster. Any code that you intercept and modify using these meta-programming tools will eventually be evaluated in the usual way, running as fast as ordinary Julia code.

You may have already used two existing examples of meta-programming in Julia:

- the  macro:

julia> @time [sin(cos(i)) for i in 1:100000]; 0.102967 seconds (208.65 k allocations: 9.838 MiB)

The  macro inserts a "start the stopwatch" command at the beginning of the code, and adds some code at the end to "stop the stopwatch", and calculate the elapsed time and memory usage. The modified code is then passed on for evaluation.

- the  macro

julia> @which 2 + 2 +(x::T, y::T) where T<:Union{Int128, Int16, Int32, Int64, Int8, UInt128, UInt16, UInt32, UInt64, UInt8} in Base at int.jl:53

This macro doesn't allow the expression  to be evaluated at all. Instead, it reports which method would be used for these particular arguments. And it also tells you the source file that contains the method's definition, and the line number.

Other uses for meta-programming include the automation of tedious coding jobs by writing short pieces of code that produce larger chunks of code, and the ability to improve the performance of 'standard' code by producing the sort of faster code that perhaps you wouldn't want to write by hand.

Quoted expressions
For meta-programming to be possible, there has to be a way for Julia to store an unevaluated but parsed expression, as soon as the parsing phase has finished. This is the ':' (colon) prefix operator:

julia> x = 3 3 julia> :x :x

To Julia, the   is an unevaluated or quoted symbol.

(If you're unfamiliar with the use of quoted symbols in computer programming, think of how quotes are sometimes used in writing to distinguish between ordinary use and special use. For example, in the sentence:

the quotes indicate that the word 'Copper' is not a reference to the metal, but to the word itself. In the same way, in, the colon before the symbol is to make you and Julia think of 'x' as an unevaluated symbol rather than as the value 3.)

To quote whole expressions rather than individual symbols, start with a colon and then enclose the Julia expression in parentheses:

julia> :(2 + 2) :(2 + 2)

There's an alternative form of the  construction that uses the   ... keywords to enclose and quote an expression:

which returns:

quote #= REPL[123]:2 =# 2 + 2 end

And this expression:

returns:

quote #= REPL[124]:2 =# for i = 1:10 #= REPL[124]:3 =# println(i) end end

The  object is of type  :

It's parsed, primed, and ready to go.

Evaluating expressions
There's also a function for evaluating an unevaluated expression. It's called :

With these tools, it's possible to create any expression and store it without having it evaluate:

returning:


 * (for i = 1:10 # line 2:

println(i) end)

and then to recall and evaluate it later:

julia> eval(e) 1 2 3 4 5 6 7 8 9 10

More usefully, it's possible to modify the contents of the expression before it's evaluated.

Inside Expressions
Once you have Julia code in an unevaluated expression, rather than as a piece of text in a string, you can do things with it.

Here's another expression:

which returns:

quote #= REPL[125]:2 =# a = 2 #= REPL[125]:3 =# b = 3 #= REPL[125]:4 =# c = 4 #= REPL[125]:5 =# d = 5 #= REPL[125]:6 =# e = sum([a, b, c, d]) end

Notice the helpful line numbers that have been added to each line of the quoted expression. (The labels for each line are added on the end of the previous line.)

We can use the  function to see what's inside this expression:

julia> fieldnames(typeof(P)) (:head, :args, :typ)

The  field is. The  field is an array, containing expressions (including comments). We can examine these with the usual Julia techniques. For example, what's the second subexpression:

julia> P.args[2] :(a = 2)

Print them out:

1: #= REPL[125]:2 =# 2: a = 2 3: #= REPL[125]:3 =# 4: b = 3 5: #= REPL[125]:4 =# 6: c = 4 7: #= REPL[125]:5 =# 8: d = 5 9: #= REPL[125]:6 =# 10: e = sum([a, b, c, d])

As you can see, the expression  contains a number of sub-expressions. We can modify this expression quite easily; for example, we can change the last line of the expression to use  rather than , so that, when P is evaluated, it will return the product rather than the sum of the variables.

julia> eval(P) 14 julia> P.args[end] = quote prod([a,b,c,d]) end quote                   #= REPL[133]:1 =#     prod([a, b, c, d])  end                   julia> eval(P) 120

Alternatively, you can target the  symbol directly by burrowing into the expression:

julia> P.args[end].args[end].args[1] :sum julia> P.args[end].args[end].args[1] = :prod :prod julia> eval(P) 120

The Abstract Syntax Tree
This way of representing your code once it's been parsed is referred to as the AST (Abstract Syntax Tree). It's a nested hierarchical structure that's designed to allow both you and Julia to easily process and modify the code.

The very useful  function lets you easily visualise the hierarchical nature of an expression. For example, the expression  is represented like this:

julia> dump(:(1 * sin(pi/2)))

Expr head: Symbol call args: Array{Any}((3,)) 1: Symbol * 2: Int64 1 3: Expr head: Symbol call args: Array{Any}((2,)) 1: Symbol sin 2: Expr head: Symbol call args: Array{Any}((3,)) 1: Symbol / 2: Symbol pi           3: Int64 2 typ: Any typ: Any typ: Any

You can see that the AST consists entirely of Exprs and atoms (e.g. symbols and numbers).

Expression interpolation
In a way, strings and expressions are similar — any Julia code they happen to contain is usually unevaluated, but you can have some of the code evaluated using interpolation. We've met the string interpolation operator, the dollar sign ($). When used inside a string, and possibly with parentheses to enclose the expression, this evaluates the Julia code and inserts the resulting value into the string at that point:

julia> "the sine of 1 is $(sin(1))" "the sine of 1 is 0.8414709848078965"

In just the same way, you can use the dollar sign to include the results of executing Julia code interpolated into an expression (which is otherwise unevaluated):

julia> quote s = $(sin(1) + cos(1)); end

quote # none, line 1: s = 1.3817732906760363 end

Even though this is a quoted expression and hence unevaluated, the value of  was calculated and inserted into the expression, replacing the original code. This operation is called "splicing".

As with string interpolation, the parentheses are needed only if you want to include the value of an expression — a single symbol can be interpolated using just a single dollar sign.

Macros
Once you know how to create and handle unevaluated Julia expressions, you'll want to know how you can modify them. A  is a way of generating a new output expression, given an unevaluated input expression. When your Julia program runs, it first parses and evaluates the macro, and the processed code produced by the macro is eventually evaluated like an ordinary expression.

Here's the definition of a simple macro that prints out the contents of the thing you pass to it, and then returns the expression to the calling environment (here, the REPL). The syntax is very similar to the way you define functions:

You run macros by preceding the name with the  prefix. This macro is expecting a single argument. You're providing unevaluated Julia code, you don't have to enclose it with parentheses, like you do for function arguments.

First, let's call this with a single numeric argument:

julia> @p 3 3

Numbers aren't expressions, so the  condition inside the macro didn't apply. All the macro did was return. But if you pass an expression, the code in the macro has the opportunity to inspect and/or process the expression's content before it is evaluated, using the  field:

julia> @p 3 + 4 - 5 * 6 / 7 % 8 Any[:-,:(3 + 4),:(((5 * 6) / 7) % 8)] 2.7142857142857144

In this case, the  condition was triggered, and the arguments of the incoming expression were printed in unevaluated form. So you can see the arguments as an array of expressions after being parsed by Julia but before being evaluated. You can also see how the different precedence of arithmetic operators has been taken into account in the parsing operation. Notice how the top-level operators and subexpressions are quoted with a colon.

Also notice that the macro  returned the argument, which was then evaluated, hence the. But it doesn't have to — it could return a quoted expression instead.

As an example, the built-in  macro returns a quoted expression rather than using   to evaluate the expression inside the macro. The quoted expression returned by  is evaluated in the calling context when the macro has done its work. Here's the definition:

Notice the  expression. This is the way that you 'escape' the code you want to time, which is in, so that it isn't evaluated in the macro, but left intact until the entire quoted expression is returned to the calling context and executed there. If this just said, then the expression would be interpolated and evaluated immediately.

If you want to pass a multi-line expression to a macro, use the  ... form:

Any[:( # none, line 2:),:((2 + 2) - 3)] 1

(You can also call macros with parentheses similar to the way you do when calling functions, using the parentheses to enclose the arguments:

julia> @p(2 + 3 + 4 - 5) Any[:-,:(2 + 3 + 4),5] 4

This would allow you to define macros that accepted more than one expression as arguments.)

and
There's an  function, and an   macro. You might be wondering what's the difference between the two?

julia> ex = :(2 + 2) :(2 + 2) julia> eval(ex) 4 julia> @eval ex :(2 + 2)

The function version expands the expression and evaluates it. The macro version doesn't expand the expression you supply to it automatically, but you can use the interpolation syntax to evaluate the expression and pass it to the macro.

julia> @eval $(ex) 4

In other words:

julia> @eval $(ex) == eval(ex) true

Here's an example where you might want to create some variables using some automation. We'll create the first ten squares and ten cubes, first using :

which creates a load of variables named, such as:

julia> var_squares_5 25

and then using :

which similarly creates a load of variables named, such as:

julia> var_cubes_5 125

Once you feel confident, you might prefer to write like this:

julia> [@eval $(Symbol("var_squares_$(i)")) = ($i^2) for i in 1:10]

Scope and context
When you use macros, you have to keep an eye out for scoping issues. In the previous example, the  syntax was used to prevent the expression from being evaluated in the wrong context. Here's another contrived example to illustrate this point.

This macro declares a variable, and returns a quoted expression containing   and an escaped version of.

Now, outside the macro, declare a symbol :

julia> s = 0

Run the macro:

julia> @f 2 (4,0)

You can see that the macro returned different values for the symbol : the first was the value inside the macro's context, 4, the second was an escaped version of , that was evaluated in the calling context, where   has the value 0. In a sense,  has protected the value of   as it passes unharmed through the macro. For the more realistic @time example, it's important that the expression you want to time isn't modified in any way by the macro.

Expanding macros
To see what the macro expands to just before it's finally executed, use the  function. It expects a quoted expression containing one or more macro calls, which are then expanded into proper Julia code for you so that you can see what the macro would do when called.

julia> macroexpand(Main, quote @p 3 + 4 - 5 * 6 / 7 % 8 end) Any[:-,:(3 + 4),:(((5 * 6) / 7) % 8)] quote #= REPL[158]:1 =# (3 + 4) - ((5 * 6) / 7) % 8 end

(The  is a filename and line number reference that's more useful when used inside a source file than when you're using the REPL.)

Here's another example. This macro adds a  construction to the language.

This is used as follows:

julia> @dotimes 3 println("hi there") hi there hi there hi there

Or, less likely, like this:

julia> @dotimes 3 begin      for i in 4:6                   println("i is $i")     end                      end                       

i is 4 i is 5 i is 6 i is 4 i is 5 i is 6 i is 4 i is 5 i is 6

If you use  on this, you can see what happens to the symbol names:

with the following output:

The  local to the macro itself has been renamed to , so as not to clash with the original   in the code we passed to it.

A more useful example: @until
Here's how to define a macro that is more likely to be useful in your code.

Julia doesn't have an until condition ... do some stuff ... end statement. Perhaps you'd like to type something like this:

You'll be able to write your code using the new  macro like this:

but, behind the scenes, the work will be done by actual code with the following structure:

This forms the body of the new macro, and it will be enclosed in a  ... block, like this, so that it executes when evaluated, but not before:

So the nearly-finished macro code is like this:

All that remains to be done is to work out how to pass in our code for the  and the   parts of the macro. Recall that  allows code to pass through 'escaped' (i.e. unevaluated). We'll protect the condition and block code from being evaluated before the macro code runs.

The final macro definition is therefore:

The new macro is used like this:

julia> i = 0 0 julia> @until i == 10 begin             global i += 1                          println(i)                 end                      

1 2 3 4 5 6 7 8 9 10

or

julia> x = 5 5 julia> @until x < 1 (println(x); global x -= 1) 5 4 3 2 1

Under the hood
If you want a more complete explanation of the compilation process than that provided here, visit the links shown in Further Reading, below.

Julia performs multiple 'passes' to transform your code to native assembly code. As described above, the first pass parses the Julia code and builds the 'surface-syntax' AST, suitable for manipulation by macros. A second pass lowers this high-level AST into an intermediate representation, which is used by type inference and code generation. In this intermediate AST format all macros have been expanded and all control flow has been converted to explicit branches and sequences of statements. At this stage the Julia compiler attempts to determine the types of all variables so that the most suitable method of a generic function (which can have many methods) is selected.