ExToErl - Convert Elixir expressions into Erlang

Your Libraries & OS Mentoring Libraries
1

Docs here: https://hexdocs.pm/ex_to_erl/ExToErl.html (with lots of examples!)
Github repo here: https://github.com/tmbb/ex_to_erl

While working on my (still mostly vaporware) mutation testing framework, I found myself working with Erlang abstract code generated by the Elixir compiler. It’s useful to have short snippets for testing purposes, and it turns out it’s very hard to convert an Elixir expression into the Erlang equivalent. It’s actually very easy to get the abstract code of a full module compiled by the Elixir compiler but there is no easy way of getting the result of a single expression such as a + b.

I’ve decided to isolate the functionality that does the conversion into its own package so that others might be able to play with it, and in the process learn a bit more about the code the Elixir compile emits. This is meant to be used as a learning tool and not as part of a serious application that runs in production.

Examples

Convert Elixir source into erlang source:

iex> ExToErl.elixir_source_to_erlang_source("a")
      "_a@1\n"

      iex> ExToErl.elixir_source_to_erlang_source("a + b")
      "_a@1 + _b@1\n"

      iex> ExToErl.elixir_source_to_erlang_source("a + b < f.(x)")
      "_a@1 + _b@1 < _f@1(_x@1)\n"

      iex> ExToErl.elixir_source_to_erlang_source("a or b") |> IO.puts()
      case _a@1 of
        false -> _b@1;
        true -> true;
        __@1 -> erlang:error({badbool, 'or', __@1})
      end

      :ok

      iex(3)> ExToErl.elixir_source_to_erlang_source("a.b") |> IO.puts()
      case _a@1 of
        #{b := __@1} -> __@1;
        __@1 when erlang:is_map(__@1) ->
            erlang:error({badkey, b, __@1});
        __@1 -> __@1:b()
      end

      :ok

Convert elixir source into erlang abstract code:

Single expressions:

  iex> ExToErl.elixir_source_to_erlang_abstract_code("a + b")
  {:op, 1, :+, {:var, 1, :_a@1}, {:var, 1, :_b@1}}

  iex> ExToErl.elixir_source_to_erlang_abstract_code("a <= b")
  {:op, 1, :"=<", {:var, 1, :_a@1}, {:var, 1, :_b@1}}

Elixir blocks (only the last expression is returned):

  iex> ExToErl.elixir_source_to_erlang_abstract_code("_ = a + b; c + d")
  {:op, 1, :+, {:var, 1, :_c@1}, {:var, 1, :_d@1}}

You can import functions and macros inside your Elixir expression:

  iex> ExToErl.elixir_source_to_erlang_abstract_code("import Bitwise; a >>> b")
  {:op, 1, :bsr, {:var, 1, :_a@1}, {:var, 1, :_b@1}}

  iex> ExToErl.elixir_source_to_erlang_abstract_code("import Bitwise; a &&& b")
  {:op, 1, :band, {:var, 1, :_a@1}, {:var, 1, :_b@1}}

Some expressions may raise warnings, although they should be the same wanings
as if the Elixir expression were to be compiled inside a normal Elixir module:

  iex> ExToErl.elixir_source_to_erlang_abstract_code("a = b")
  warning: variable "a" is unused

  warning: variable "a" is unused

  {:match, 1, {:var, 1, :_a@2}, {:var, 1, :_b@1}}

Some Elixir operators are actually macros or special forms which can be expanded
into quite complex Erlang code:

  iex> ExToErl.elixir_source_to_erlang_abstract_code("a or b")
  {:case, 1, {:var, 1, :_a@1},
  [
    {:clause, [generated: true, location: 1], [{:atom, 0, false}], [],
      [{:var, 1, :_b@1}]},
    {:clause, [generated: true, location: 1], [{:atom, 0, true}], [],
      [{:atom, 0, true}]},
    {:clause, [generated: true, location: 1], [{:var, 1, :__@1}], [],
      [
        {:call, 1, {:remote, 1, {:atom, 0, :erlang}, {:atom, 1, :error}},
        [{:tuple, 1, [{:atom, 0, :badbool}, {:atom, 0, :or}, {:var, 1, :__@1}]}]}
      ]}
  ]}

For more examples, read the docs in the link above.

Implementation details

This package works by doing scary things in the BEAM runtime. First, it generates a dummy Elixir module with a single function in it; inside that function it places the elixir expression(s). Then it compiles the code into an in-memory BEAM file. Then, it extracts the erlang abstract code from the BEAM file, and if needed converts it into source code using functions from the Erlang standard library.

Because it compiels Elixir code, it generates atoms at runtime. It’s possible that if you try to convert expressions concurrently there will be race conditions (I have to fix it by making everything go through a pool of genservers so that no two processes try to compile the same module at the same time; I have no idea which protections are in place to avoid this happening…)

13 Likes
2

This is very cool!

When doing this conversion, is the Erlang code that gets returned decompiled from the BEAM AST before or after (most) BEAM optimizations have been applied?
I would see a use-case to easily check in which ways the BEAM is able to optimize (or not) certain expressions. :smiley:

2 Likes
3

It’s the Erlang code before all optimizations. All optimizations happen somewhere between the “Core Erlang” pass and the BEAM bytecode.

2 Likes
4

It is also useful to be able to convert Erlang code to Elixir. It is actually not too hard… After all, you can just translate the constructs one by one, so as soon as you can handle all of Erlang’s abstract code, you’re done. But if we’re converting Erlang code that was generated by the Elixir compiler, we can do better and preserve some of the macros and special forms that were used to generate the erlang code. Also, one can assume that variable names of the form _<valid_elixir_variable_name>@<integer> is the same as the elixir <variable_name>.

Converting Erlang back to Elixir is also useful for error reporting in my Darwin mutation framework. The only question is: should the erlang_to_elixir converter be part of Darwin or moved here? I might make compromises that are acceptable in Darwin but not acceptable in a “general purpose” library…

1 Like
5

What kinds?

6

There is an experimentel erlang to elixir converter, which doesn’t support all of Erlang’s syntax yet.

For example, when my translator finds something it can’t translate, it returns ..., which is valid Elixir AST and a nice filler. For example, I don’t support maps yet, so I get the following code:

iex(7)> abstract_code = ExToErl.erlang_source_to_abstract_code(~S[module:f(#{"key" => value}).])
{:call, 1, {:remote, 1, {:atom, 1, :module}, {:atom, 1, :f}},
 [{:map, 1, [{:map_field_assoc, 1, {:string, 1, 'key'}, {:atom, 1, :value}}]}]}
iex(8)> elixir_ast = ErlToEx.erl_to_ex(abstract_code)
{{:., [], [:module, :f]}, [], [{:..., [], ErlToEx}]}
iex(9)> Macro.to_string(elixir_ast)
":module.f(...)"

The idea of using ... as a filler is good for Darwin, but maybe for other applcations it should raise an error or return {:ok, success} | :error}. But if I make it complete (and there’s no reason why I can’t make it complete) there won’t be any problems.

7

I’d say make it generic and add it to this this, trying to get full completion is definitely the best goal and can get more help. :slight_smile:

8

There is https://github.com/olafura/beam_to_ex for erlang to elixir. It doesn’t support some important things like records but I’m getting close to be as feature complete as I can be

1 Like
9

It looks like your library:

  1. Doesn’t provide an easy API to convert isolated Erlang expressions and is meant to be used to convert entire modules only

  2. You convert Erlang operators into Elixir operators in a way that changes the semantics. For example, you convert _a@1 andalso _b@1 into a and b. Actually, Elixir compiles a and b into a complex case expression which gives you a different error message. You also don’t seem to recognize the complex case expression as the macroexpanded and operator. The same goes for the other boolean operators. The same goes for if statements.

Is this true? If so, then I’m afraid I can’t use your library and should write my own instead. It doesn’t make your library any worse (I mean, point 2 kinda does, because it would be cool to invert the compilation of the boolean operators), but it means it’s optimized for things different from what I’m doing.

1 Like
10

For the first point beam_to_elixir_ast is meant as a library while beam_to_elixir is the command line part. You can just use BeamToExAst.convert/2 to convert the Erlang expressions to Elixir.

The second point is something I hadn’t noticed, but a really good observation. I’ve been working on this library for a couple of years, mostly because I haven’t needed to use it and can’t give it all the time I want. If you know of any more issues like this then please file an issue, I would love some more test cases.

11

I think I had written the andalso to and because of:

But of course and is more complex. I support adding information for your children so I should be able to match the correct behaviour if it doesn’t change :smiley:

I also want to add a lot more just plain erlang tests since I think I can’t match everything converted to Erlang expressions and back to Elixir. Since that is just a corner case anyway.

12

That’s only for in guards, look at the nil -> part for the usual case.

13

Yeah I have to add some tests to make sure I’m not doing anything silly. I hope this will also work for Erlang behaviours otherwise I have to figure out what to do with edge cases for andalso

1 Like