How to get an AST and Macro.Env of a function by name and arity?

Questions / Help
, ,
1

Summary

Is there any way to write a function which can be invoked in compile time and which can return an AST and Macro.Env of given function.

Example

We have a module

defmodule MMM do
  def fff(x), do: x + 1
end

And we can execute some function in compile time which will work like

iex> Macro.get_ast(MMM, :fff, 1)
{
 {:def, [context: Elixir, import: Kernel],
  [
    {:f, [context: Elixir], [{:x, [], Elixir}]},
    [do: {:+, [context: Elixir, import: Kernel], [{:x, [], Elixir}, 1]}]
  ]},
  %Macro.Env{}
}

PS

I know there is no such function, but how can something like this be implemented? What problems should I expect while implementing this function?

1 Like
2

Such a function like that doesn’t exist because the AST is long long gone at that point. You could always make your own macro, say defmodulex or so and just use it instead of defmodule and have it build a new hidden function called __ASTS__/1 where you pass it an option of what to return and it can return the AST for whatever you asked for. Basically compile the compile and add in that function that just returns the AST of the module that was passed in?

Better question though, ‘why’ do you want it?

1 Like
3

I am trying to investigate ability to perform some simple automatic function inlining optimizations in compile time
I thought about defmodulex-style approach, but this looks ugly and can’t be acheived with functions in dependencies

I thought that there might be some way to override :elixir_module module (unload it before compilation and load modified version which stores AST per module), or create some compiler tracer, which will try to store all functions in some ets or process (but I don’t know if tracers are capable of doing this)

1 Like
4

You actually can, but this is definitely more work… ^.^;

First task would be to fork that file, then make the changes you want, then load your ‘loader’ module first and replace the built-in module with that by forcing it to load via a call in the Code module (forget the name but it’s not hard to figure out).

Traces can’t really do that no, well, maybe if you hook the file loading but then you’d have to do a whole lot more work, so not recommended… ^.^;

EDIT1: Are you sure that inlining functions is really that necessary with the new OTP-24 JIT?

EDIT2: Also, what if a module gets hot-replaced, how will you update your inlined versions?

1 Like
5

As far as I know, OTP 24 JIT performs just beam-to-native translations without any optimizations

This is a good question. Right now I am trying to acheive private functions optimizations. (I know that they can be inlined with @compile {:inline, [func: arity]}, I am just trying to implement basic optimization pipeline).

However, hot reloading (with IEx.Helpers.recompile, for example) forces all modules calling replaced module (and their callers and so on) to be recompiled too. So tracking updates of reloaded inlined functions won’t be a difficult problem

Thanks for this idea, I will try to look into it and update this thread with my founding!

1 Like
6

Private functions are already heavily inlined. Do you mean public functions that are ‘private’ to your library (which aren’t actually private)?

That’s only useful in IEx, that’s not normal runtime hot code swapping, and even then it won’t always unless force: true is used. I’m more referencing release upgrade hotswapping as that would be an exceptionally very bad time to get bugs from aggressive out-of-compiler inlining.

Just be careful with it, there’s a lot of dragons in those seas!

I still highly recommend using macro’s instead of trying to inline, macro’s naturally inline and only what you make as them. Often shorter functions can be more efficient however (in the same module especially, which can also be made by macro’s).

1 Like
7

I ment private-private (defp). I didn’t knew that they are heavily inlined. Where can I read about it?

That’s true, but this code can be scraped off for hot-swapping in releases

1 Like
8

this is possible using Elixir compiler tracer hooks. I am pretty sure it broadcasts the AST and __ENV__ of a function being compiled.

https://hexdocs.pm/elixir/Code.html#module-compilation-tracers

Worst case scenario, collect em’ all, serialize them using :erlang.term_to_binary, and then stash everything into some file in /priv

3 Likes
9

Oh hey, elixir has a hook? Interesting, beam doesn’t as far as I know, so it wouldn’t work with non-elixir code, but that could do it.

10

In the erlang side things like -compile({inline,24}). are specified to inline module-local functions with a given weight or less, 24 is the default weight, so most ‘shortish’ functions will get inlined directly and always. It can be overridden though with a different weight, like 1000 if you want to inline almost everything, or you can force inline specific functions by giving it a list.

Do note though, inlining larger can help with some optimizations, but overall it usually doesn’t and it’s best to leave it at default as the size overhead doesn’t help the speed at that point, and calling module-local functions require no indirected call so their only cost is a stack frame at that point (if that’s even needed).

3 Likes