sashaafm

sashaafm

Getting each stage of Elixir's compilation all the way to the BEAM bytecode

Piggy backing a bit on @dvcrn topic BEAM optimization for functions with static return type?, I’ve been trying to understand in a deeper manner how Elixir works internally to generate the BEAM’s bytecode. After reading way too many blog posts I’ve found some things:

  1. People either think that Elixir compiles directly to Erlang source code (.erl)
  2. Or people think that Elixir compiles directly to BEAM bytecode (.beam)

Both of these assumptions seem to be wrong. From the Elixir/Erlang Crash Course from Elixir’s official webpage, we can see that:

Elixir compiles into BEAM byte code (via Erlang Abstract Format).

Steps from Elixir source code to BEAM bytecode

So it’s not directly to Erlang source code, but it’s also not directly to BEAM bytecode. It is first transformed into Erlang Abstract Format (EAF). Continuing further into this topic, I’ve found a couple of blog posts, this one in particular BEAM by Example, where the author tells us the following:

Intermediate representations:

Erlang source code → Abstract Syntax Tree (‘P’) → expanded AST (‘E’) → Core Erlang (‘to_core’) → BEAM byte-code

So Elixir is first transforming to this Abstract Syntax Tree or Expanded AST intermediate representations. It should be something like this:

Elixir → Erlang Abstract Format → Core Erlang → BEAM bytecode


Note

I’ve also seen one or two posts online talking about Elixir being transformed into Erlang Forms. I’ve got no idea if these “Erlang Forms” are the same as one of the steps above or if they are an entirely different thing.


Now we’ve got a few different cases:

  • Elixir → EAF

This can be achieved through the :elixir Erlang module, that can be found here, like so:

expr = Macro.to_string(quote do: 1 + 2)
env  = :elixir.env_for_all([])
eaf  = :elixir.quoted_to_erl(expr, env)
# => Erlang Abstract Format of the quoted expression
  • EAF → Core Erlang
    and
  • Core Erlang → BEAM bytecode

I haven’t found a way to achieve these two steps. The further I’ve got is that Erlang’s compiling function can be used to get the various formats:

c(<file_name>, <format>)
c("file.erl", 'P')
c("file.erl", 'E')
c("file.erl", to_core)
c("file.erl", to

*BEAM Bytecode → Disassemble

This can be done either by c("file.erl, 'S'). or :beam_disasm.file/1, which I believe are the same thing, as far as I could find.

Example Gist

I’ve built this small Gist to better show the steps from an Erlang source code all the way to the disassembled bytecode.


Note

James Fish also spoke to me on Slack and told me to check out the :compile.forms/1 Erlang function. I don’t fully understand what this function actually does or returns. It seems to receive Erlang Abstract Format as an argument.


Erlang docs are sparse and usually scattered all around. I’ve only managed to gather some info about this topic from several sources, but I’d like to better understand this process of Elixir → BEAM. I’ve watched dozens of Elixir talks, but I don’t recall ever seeing this explained.

I’m hoping someone around here has some further knowledge on this :slight_smile:

Most Liked

josevalim

josevalim

Creator of Elixir

Yes. To be more precise, instead of “Elixir”, you could have: Elixir Source Code → Elixir Macro Expansion → Erlang Abstract Format → …

16
Post #8
rvirding

rvirding

Creator of Erlang

So a quick answer here (more later) is the erlang compiler has 2 main entry points: :compile.file which compiles a text file; and :compile.forms which takes a list of pre-parsed forms. As you have seen you can specify how “far” in the compilation you want to go, whether to pre-expanded macros and parse transforms, core erlang, kernel erlang or just the BEAM instructions (without generating a .beam file). Try doing to_kernel, dkern and dlife for some more fun.

You can also specify what type the input should be, a little anyway, so for example the option :from_core means that the input, whether file of forms, is core erlang. This is what I use in the LFE compiler where I generated core erlang forms (there AST anyway) which I then compile with :compile.forms(forms, [:from_core|options]). I found this easier than generating erlang AST.

Almost all optimisation in the compiler is done on core erlang so I don’t “lose” anything by entering there.

The c("file.erl", 'S') compiles the file only to the BEAM instructions and prints them to a .S file while :beam_diasm:file/1 looks at the beam file and disassembles it. There is also a way which I can’t remember now where you can disassemble the actual code installed in the BEAM itself. This will be slightly different from what the other two give you as there is quite a lot of optimisation done at load time.

Most of the time you don’t need to know this except for the expansion of macros and parse transforms, but it is fun. About 12 min into my talk mentioned there is a slide on the passes of the compiler. It is a bit hard to see. IIRC there is another talk I gave on about the same thing where it is easier to see the slides.

13
Post #5
ibgib

ibgib

In case you haven’t seen it, here is another resource I think would interest you on this topic: “Implementing Languages on the BEAM” with @rvirding

I’m only half-way through so far, but it talks about the intermediate steps of any language (not just Elixir) that runs on top of the BEAM(!).

10
Post #2

Where Next?

Popular in Discussions Top

pillaiindu
In django there is a cache framework backed by memcached. Rails also puts a lot of emphasis on caching, and even the idea of russian-doll...
New
Ankhers
Just a little information upfront. Generally speaking, if I feel like I need to either break a pipe chain or use an anonymous function in...
New
Crowdhailer
I’ve been hearing much about the new formatter and it’s something I have been keen to try. I find examples buy far the most illuminating...
248 19327 150
New
PragTob
Hey everyone, this has been on my mind for some time and I’d love your input on it! TLDR: I feel like maps are superioer for storing and...
New
rms.mrcs
A couple of days ago I was discussing with a friend about different approaches to write microservices. He said that if he was going to w...
New
AstonJ
If so I (and hopefully others!) might have some tips for you :slight_smile: But first, please say which area you’re finding most challen...
New
Owens
Hello all, I am developing a new mobile app with Flutter frontend and Phoenix backend. The mobile app has real-time task management and c...
New
fireproofsocks
This is more of a general question, but I’m wondering how other people in the community think about the pattern matching in function sign...
New
axelson
Decided against including more info in the title, but the gist is that Plataformatec sponsored projects will continue with the assets bei...
New
Markusxmr
Since Drab has been developed for a while in the open, introducing the Liveview functionality in a way it happend appears to undermine th...
New

Other popular topics Top

hariharasudhan94
lets say i have a sample like a = 20; b = 10; if (a &gt; b) do {:ok, "a"} end if (a &lt; b) do {:ok, b} end if (a == b) do {:ok, "equa...
New
Harrisonl
We have an ECS cluster with 4 services, where each task joins a single cluster, via discovery ECS discovery service. Currently when I de...
New
marius95
Hello everyone, I try to use an Javascript Event Handler in my root.html.leex file. Therefore I created a function in the app.js file: ...
New
baxterw3b
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
New
Patoshizzle
After calling mix ecto.create I get this error: 17:00:32.162 [error] GenServer #PID&lt;0.412.0&gt; terminating ** (Postgrex.Error) FATAL...
New
ovidiubadita
Hey all, I discovered Elixir and I love it. I always wanted to learn a functional programming and I intended to go for Haskell, but afte...
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
joaquinalcerro
Hi there, I am working with Ecto-Postgresql and I need to call all of the records from a specific table but the table has 40,000 records...
New
AstonJ
We’ve put together this wiki for Phoenix LiveView - please feel free to add any info you feel is worth including. What is Phoenix LiveV...
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New

Latest on Elixir Forum

Elixir Forum

We're in Beta

About us Mission Statement