grych

grych

Creator of Drab

Find out unbound variables in the AST, or How I Made Drab Better But Slower, or I Need A Community Opinion on the Approach

Dear Alchemists,
Assume I have an AST fragment with some unbound variables:

iex(1)> expr = quote do: IO.puts(x)
{{:., [], [{:__aliases__, [alias: false], [:IO]}, :puts]}, [],
 [{:x, [], Elixir}]}

what I want to archive is to find all unbound variables there, so have something like:

iex> unbound_vars(expr)
[:x]

or at least find out if there are any:

iex> has_unbound_vars(expr)
true

The easiest thing would be to eval the expression and catch the CompileError:

iex(2)> Code.eval_quoted expr
warning: variable "x" does not exist and is being expanded to "x()", please use parentheses to remove the ambiguity or change the variable name
  nofile:1

** (CompileError) nofile:1: undefined function x/0
    (elixir) lib/code.ex:493: Code.eval_quoted/3

But it does not look very idiomatic, and also generates a warning, which is a big deal for me, as this will be used in a library (drab), so the warning may confuse the library users as it will appear during the normal usage.

I am thinking about Macro.prewalk:

  • collect all variables from {atom, _, Elixir}
  • collect pattern matches {:=, _, [left, _]} and take all variables ({atom, _, Elixir}) from left
  • compare those lists

But I am not sure it is enough, I have a feeling I am missing something. Is the pattern match the only way to bind a variable? Are the variables always are like {atom, _, Elixir}?


Background, or I Need Your Opinion On The Approach

This is not an academic issue :slight_smile: In Drab, I collect the expressions from the template, which contain Phoenix assigns (<%= function(@assign1)%>) during the compile time, and re-evaluate them in case user wants to change it on the page, live, with poke socket, assign1: new_value.

It works well in the most cases, but crashes when using local variables defined outside of the block:

<%= for u <- @users do %>
  <%= if u != @user do %>
    <%= u %> <br>
  <% end %>
<% end %>

All works OK when you re-evaluate the for expression with poke socket, users: [...]. But when trying to poke socket, user: ... system tries to re-evaluate only the if expression and it finished with CompileError.
This may be worked around with poking both assigns in the same time with poke users: [...], user: ... (in this case Drab knows that there is no point to refresh if, as the whole block will be re-evaluated), but it is not very convenient. And confusing the users. And does not solve all the cases.

I’ve developed the alternative, much easier approach, to archive the same: instead of evaluating this partial expressions, on each poke Drab renders the whole Phoenix template (using update assigns), parses the html and pushes the fragment to the browser. This is much better from compatibility point of view, but way slower. For example, on Drab demo page, which is already using this approach, all pokes are now about 50ms slower than it was before. You can feel it in the counter example.

So, in my opinion, the best would be to combine those two approaches. When the expression has some unbound variables (we can check it in the compile time for better runtime performance), Drab would render the whole template. Otherwise, use the faster way.

Or maybe I’m wrong, and 50ms overhead of rendering template on each poke is not a big deal? In this case I could remove probably about 50% of Drab.Live.EExEngine code, making it way more stable and reliable.

The second approach simply removes all Drab.Live limitations. This is why I really like it. But the first is faster. HELP!

Oh No! Another Approach!

BTW while writing this long (sorry!) post, I realized that I could use the custom EEx marker to let the developer choose the approach! So there would be three markers:

  • <%= expression %> would work with safe, slow but compatible way - re-render the whole template
  • <%| expression %> the fast way
  • <%/ expression %> do not drab this expression

Thus, when you know what you’re doing, you may choose the faster way, but by default it would just work without any limitations. Drab is made also for beginners, and I want it to me the most intuitive as it is possible. But if you know how it works, you may try to choose better performance and write your code differently.

Most Liked

christhekeele

christhekeele

Variables in AST always take the form {name, meta, context} when is_atom(name) and is_list(meta) and is_atom(context); furthermore they are unique in that shape so the guard above will only ever catch actual variables.


If you take this approach you will find yourself re-implementing a pattern-matching lexer trying to determine which variables are bound or unbound; remember matches can be complicated and do deep destructuring.

What you really want to do is pass your code through the Elixir compiler without evaling anything. You can totally do this since the Elixir compiler is written in erlang so we can call into its AST expansion functions. This approach is far superior because it will also take into account things like in-scope variables, aliases, and imports in the provided Macro.Env (probably the __CALLER__ in your macro); the downside is while you will be consulting the single source of truth about Elixir’s variable binding, you will be calling into a somewhat private API not meant for use outside of Elixir core which is liable to change version to version.

If you want to get started down that path, I’d check out the Kernel.Utils.defguard implementation which does something similar: extracts variable references from the AST and inserts them into the ENV, so it can expand the code and get an updated env without hitting a CompileError. I imagine if you manipulate the env just right pre-expansion you can introspect on the returned post-expansion env to deduce which variables are still unbound.

josevalim

josevalim

Creator of Elixir

I want to add that this is actually hard to do correctly in practice. You would effectively need to implement a large chunk of the compiler. For example, take this code:

foo(a + b)

How many variables are used?

Well, if “foo” is a macro, then foo may define several variables in it. You would have to expand all macros recursively to give the correct answer.

For example, what about this code?

destructure [foo, bar], [1]

This is valid Elixir code today and you know that foo and bar are variables only by expanding the macro and then traversing all of Elixir special forms and reimplementing how they handle variables. The reason it works for defguard is because guards are a very small subset of the language. So reimplementing it is trivial.

qhwa

qhwa

Right, macros defined in snippets are not supported. Fortunately, I can ignore that in my library because all macros are supposed only to be introduced via a helper module outside of the snippet.

Also, I thought about another approach of taking advantage of the Elixir compiler to check unbound variables. It works by injecting binding variables until the compiler stops raising such errors. I created a poc for it. The code looks like this:

def extract_vars(ast, env \\ %Macro.Env{}, vars \\ MapSet.new())

def extract_vars(ast, env, vars) do
  try do
    Module.create(
      @temp_module,
      def_run(ast, vars),
      env
    )

    MapSet.to_list(vars)
  rescue
    err in CompileError ->
      with %CompileError{description: err_msg} <- err,
           {:ok, var} <- unbound_var_from_err_msg(err_msg),
           false <- MapSet.member?(vars, var) do
        extract_vars(ast, env, MapSet.put(vars, var))
      else
        _ -> reraise(err, __STACKTRACE__)
      end
  end
end

# It defines a function `run(binding)`, looking like this:
#
# def run(binding) do
#   a = Keyword.fetch!(binding, :a) # <- injected variable definition
#   b = Keyword.fetch!(binding, :b) # <- injected variable definition
#   ...
#   # here goes the snippet
#   # ...
# end
defp def_run(ast, args) do
  quote do
    def run(binding) do
      unquote(def_args(args))
      unquote(ast)
    end
  end
end

defp def_args(args) do
  for arg <- args do
    quote do
      unquote(Macro.var(arg, __MODULE__)) = Keyword.fetch!(binding, unquote(arg))
    end
  end
end

defp unbound_var_from_err_msg(err_msg) do
  reg = ~r/undefined (function|variable) \^?(?<var>\w+)/

  case Regex.named_captures(reg, err_msg) do
    %{"var" => name} ->
      {:ok, String.to_atom(name)}

    _ ->
      :error
  end
end

Basically, it works with some small issues such as concurrent compiling, which is not difficult to fix IMO.
However, I don’t know if this is a good idea because it feels hacky :smiley: .

Where Next?

Popular in Questions Top

marius95
Hello everyone, I try to use an Javascript Event Handler in my root.html.leex file. Therefore I created a function in the app.js file: ...
New
aadeshere1
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
New
chokchit
** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 2733ms. You can configure how long re...
New
lessless
I believe there are people here who are dealing with CSV files import on the daily basis, and since Excel is a really popular tool there ...
New
dokuzbir
I want to highlight html closing tags when i click a html tag. That works in .html files but doesnt work for html.eex templates. How can...
New
aalberti333
As the title describes, I’m trying to run Enum.map() over a list of key/value pairs, where the value is a map. My data looks like this: ...
New
LegitStack
I’m trying to make a websocket server in Phoenix or raw Elixir. I heard about gun, I think I could use cowboy, but since I’m not that sma...
New
bsollish-terakeet
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
New
chensan
I have a User schema with a :from_id field set to type :string: defmodule TweetBot.Repo.Migrations.CreateUsers do use Ecto.Migration ...
New
WestKeys
Currently suffering from paralysis by [HTTP client] analysis. This is rather unusual in Elixirland as there tends to be consensus on the ...
New

Other popular topics Top

chrismccord
As promised, the first release candidate of Phoenix 1.3.0 is out! This release focuses on code generators with improved project structure...
New
AstonJ
Posting this to see if we can make things easier for people to get into Neovim. If you use Neovim and have a favourite distro please let ...
New
stefanluptak
Hello everybody, usually, I use a 29" ultra-wide monitor for VSCode which can easily accomodate explorer (files panel) + file with code ...
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
Emily
I have VueJS GUIs with the project generated using Webpack. I have Elixir modules that will need to be used by the VueJS GUIs. I fore...
New
Lily
In templates/appointment/index.html.eex: &lt;%= for appointment &lt;- @appointments do %&gt; &lt;tr&gt; &lt;td&gt;&lt;%= appoi...
New
fayddelight
I tried installing elixir 1.11.2 erlang 23.3.4 via asdf in my zsh shell. Enabled the versions locally and globally. When I list them ...
New
hariharasudhan94
lets say i have a sample like a = 20; b = 10; if (a &gt; b) do {:ok, "a"} end if (a &lt; b) do {:ok, b} end if (a == b) do {:ok, "eq...
New
baxterw3b
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
New
axelson
This post is a wiki (feel free to hit the edit button near the bottom right of this post to add your own changes!) This post collects co...
239 47849 226
New

We're in Beta

About us Mission Statement