Today I realized how powerful macros really are

This morning, I came to a sudden realization: Elixir’s Macro system is even more powerful than I’d thought.
My mind is blown.

Let me try to explain:

Macros take one or multiple expressions as input. These expressions are not evaluated before passing them into the macro. Instead, they are passed in as AST (an Abstract Syntax Tree). If, when and how we want to evaluate these ASTs is completely up to the macro implementation.

Observe the following example:

defmodule Symmath do
  defmodule Expr do
    defstruct [:ast]
  
    defimpl Inspect do
      def inspect(expr, opts) do
        ast_str = Macro.to_string(expr.ast)
        "Symmath.expr(#{ast_str})"
      end
    end
  end


  @doc """
  Creates a Symmath Expression.
  `expression` is not immediately evaluated.
  Instead, it is treated as a symbolic representation.
  """
  defmacro expr(expression) do
    escaped_expr = Macro.escape(expression)
    quote do 
        %Expr{ast: unquote(escaped_expr) }
    end 
  end
end

Paste this in IEx or something, and call it as follows:

require Symmath
Symmath.expr(x*2 + 3*pi = pow(x, 2))

These things that blew my mind:

  • neither x or pi have to be defined.
  • functions like pow do not actually need to be defined. They are just ‘abstract syntax’ as everyting else and we can decide what to do with it at a later time.
  • we can just use any valid Elixir operators in this expression, and change their meaning or manipulate them if we want.
  • Things like Symmath.expr(100/0) are completely valid statements.

The example above only reads the AST, puts it into a struct, and when someone inspects this struct, it shows the stringified version of the AST inside. But we could now pass these structs around, write ways to simplify these expressions, rewrite them, evaluate them (whole or partially), etc.

This opens up a whole new world for Domain Specific Languages.

I think we can build some really crazy and amazing things with this idea!

8 Likes

Yeah, it sort of becomes a game of “is this valid Elixir syntax?”. Just quote expressions and have a look. For example, pow(x, 2) is fine, but in an equation dsl, it might be nice to write x^2. Valid syntax.

1 Like

Yep, things that are special characters(of whom Elixir has very few) or attempting to use unary operators on two operands are the two things that are not possible. That basically means that # and ^ are out. Nearly all other things are fair game.

1 Like

Macros are one of those items that are conceptually simple but hard to read and utilize in practice.

Each piece is simple in isolation. Macros are special functions that receive an AST and return an AST. quote/2 turns an expression into an AST. unquote/1 evaluates an expression and injects the results into a quoted expression (a.k.a. an AST fragment). And of course you have the concept of “hygiene” - variables created in a macro are private to that macro unless explicitly defined otherwise.

So again, the pieces are, on their own, easy to wrap your head around, but putting them together I’ve found difficult.

There are two great resources out there for learning Elixir’s metaprogramming capabilities - @chrismccord’s Metaprogramming Elixir and @sasajuric’s Understanding Elixir Macros mini-series. Head still hurts, and it will probably take two readings of each to have macros really sink in for me. (Oh, and of course the documentation on quote/2 is incredible).

I’m not sure what the point of my post is/was now. :101:

5 Likes

So ^ is a bit weird.

quote do 1^2 end
** (SyntaxError) iex:1: syntax error before: '^'

quote do x^2 end
{:x, [], [{:^, [], [2]}]}

It would be ill-advised to include an operator with the gotcha that it cannot operate on a literal.

1 Like

The problem here isn’t that ^ cannot operate on a literal (indeed, statements like ^2 are allowed), but that ^ is only defined as a unary operator (only taking a single input). For + and - this isn’t the case, as they have both a unary ( -2) and a binary variant ( 1 - 2).

1 Like

Of course the obvious solution to the problem of operators is to use lisp which doesn’t have operators, and practically no reserved characters. :laughing:

1 Like

Limiting the amount of definable operators was a deliberate choice by José Valim. Operators are a nice and concise way to express things, but nobody knows what <<^ or <*> or =>= or $~! means when you first encounter them. In that way, they are very implicit. This is the reason that languages like Haskell are deemed hard to understand, because they are very operator-heavy.

I understand the choice @josevalim made here, but I do frequently miss this freedom in Elixir.

iex(server@snip.local)3> quote do lit 1,^ 2 end
{:lit, [], [1, {:^, [], [2]}]}

Eh, its ugly, but for a dsl it ‘might’ work depending on the purpose (probably not a math one)…

On a side-note, it would be freaking awesome to get some kind of defreadmacro or so, it gives you the binary/string input stream until some specified ending ‘thing’…

  # THE HORROR
  defreadmacro python(stream, env) do
    # blah
  end

  # elsewhere
  python~(
  a = 2+3
  print "blah"
  )~

Hmm, although I guess you could already do things like that…

defmacro myDSEL(input) when is_binary(input) do
  LISPToElixir.parse_to_ast(input)
end

def do_something() do
  myDSEL """
    (Enum.map (1 3.14 42) (:fn (x) (IO.inspect x)))
    """
end

/me wonders if someone is actually going to make that now…

Ah, lisp, in which every system is its own dialect of lisp :wink:

2 Likes

That just shows how versatile lisp is, you are not bound to just one.

Seriously though, any language with macros gives you this feature/plague, even elixir. You just have to show restraint.

3 Likes

feature / plague … I like that.

1 Like

@OvermindDL1 You could define a Sigil for something like that. This has the advantage that you don’t need to use "" around whatever you pass in.


I took the idea of creating a Symbolic Math library that used Elixir’s AST and started tinkering a little. The result is symmath.

It is still far from finished, and probably contains some bugs (I haven’t gotten around to writing tests yet, as the library is still in full flux), but it really is a lot of fun to work on.

Some code examples:

iex> import Symmath
iex> f = expr(1+1)
Symmath.expr(1+1)
iex> simplify(f)
Symmath.expr(2)
iex> g = expr pow(2*x-3, 3*pi)
Symmath.expr(pow(2 * x - 3, 3 * pi))
iex> g |> deriv
Symmath.expr(pow(2 * (3 * pi) - 3, 3 * pi) * (0 * pi + 3 * 1))
iex> g |> deriv |> simplify
Symmath.expr(3 * pow(6 * pi - 3, 3 * pi))

Both the simplification and the derivation can still be improved (Things involving multiple applications of the chain rule will go in an infinite loop right now, for instance). Also, there isn’t support for trigonometric functions or logarithms yet.

Still, the basic idea is there, so I’m interested about what you think :sweat_smile: .

3 Likes

I’m implementing Rex a tiny concatenative language on top of Elixir syntax, currently it’s only an experiment on just taking an Elixir quoted expression, transforming it into a post-fix ast (what I call a Rex AST). So basically if you have some elixir like 1 + 2 it gets into rex ast: 1 2 +, then this rex ast is basically transformed into a list of functions each taking a stack, so, for example the 2 function returns something like [2 | stack] and + operates on the two top level values on the stack. Currently rex is just a toy will see how it goes, and of course being on top of Elixir syntax has some particularities, be sure tu check the README if you get interested. :slight_smile: Cheers.

1 Like

@vic: Wow! That is really nice :smiley: . The usage of ~> to make concatenative values allowed syntax is a very interesting idea.

Have you seen Jux, the concatenative language that I am working on? (Which is not a subset of Elixir’s macros, but does have an interpreter built in Elixir)

I didn’t know of it but will certainly take a look at it :slight_smile: thanks for the link.