PhoenixUndeadView - let's discuss optimization possibilities for something like Phoenix LiveView

Major Changes

I’ve been more or less blinded by the idea that the EEx templates should return something that’s similar to a quoted expression. As @josevalim said some time ago, this is a mistake. I’ve lost a couple hours which I’ll never get back for not listening to his advice. This tells me I should following to his advice next time. Or not, who knows…

So this is how it will work now:

The engine will no longer compile the text into a real quoted expression. I’ll make it so that it compiles into a structure made from tuples and lists of 2-tuples. I like to think of 2-tuples as algebraic data types, and it would help if Elixir supported real algebraic data types, but well, we have to make do with what we have.

The “datatype” of the intermediate EEx templates would be the following:

{:undead_eex,
  [
    # static text
    {:static, "static binary"},
    # <%= ... %>
    {:dynamic, expr1},
    # <% ... %>
    {:dynamic_no_output, expr2}
    # <%| ... %>
    {:fixed, expr2}
    ...
  ]}

The critical thing is that I can nest an {:undead_eex, _} template inside the {:dynamic, _} constructor. This nested template could be the result of expanding a macro inside a :dynamic segment. For example, I could have:

{:undead_eex,
  [
    segment1,
    segment2,
    {:dynamic, expression_with_a_macro},
    segment3,
    segment4
  ]}

After expanding the macro (inside the UndeadEEx engine or in a postprocessing step), this could return the following template:

{:undead_eex,
  [
    segment1,
    segment2,
    {:dynamic, {:undead_eex, contents}}
    segment3,
    segment4
  ]}

I can statically simplify this into:

{:undead_eex,
  [
    segment1,
    segment2
  ] ++ contents ++ [
    segment3,
    segment4
  ]}

I am now compiling the text into such an abstract format, which preserves interesting semantic distinctions between the several types of segments (instead of a quoted expression, which kinda blends everything together in executable Elixir code).

I can apply valid algebraic transformations to this structure in order to optimize it further. The transformation above is an example (actually the only one I can think of right now). The following expressions are equivalent:

{:dynamic, {:undead_eex, contents}} # is equivalent to...
contents

It’s possible that there are other valid transformations I can apply, but the important is the idea.

This is all a bit abstract, so let me show how this can be quite useful.

A template is a 2-tuple

As I said, a compiled template is a 2-tuple of the form {:undead_eex, contents}. Templates are fed into the Engine as text files. A compiled template is the “raw” result of compiling a text file with the UndeadEngine.

This compiled template is not very useful by itself. It’s not a quoted expression which you splice into Elixir’s AST to get some executable code. You should think of it as an intermediate representation in a compiler

Templates can be nested inside each other

Because a template is simply a data structure, it’s obvious we can nest them inside each other:

{:undead_eex,
  [
    ...,
    {:dynamic, {:undead_eex,
      [
        {:dynamic, {:undead_eex, ...}}
      ]}},
      ...
  ]}

When we have nested templates, we can always apply the rule above to flatten them into a single template.

We can define reusable widgets

We can define reusable widgets as macros. Those macros will be expanded into templates (which are a data structure and not a quoted expression representing executable code!).

If you have a my_widget(arg1) macro inside a :dynamic segment, it will be expanded. If the result of the expansion is a valid compiled template (i.e. {:undead_eex, ...}), it can be flattened into the parent template.

This is great for optimization, because we can merge adjacent binaries.

(Some) Widgets in phoenix_html can be reimplemented as macros

The phoenix_html package defines a number of widgets, implemented as functions. For example:

iex> import Phoenix.HTML.Tag
Phoenix.HTML.Tag
iex> tag(:input, [id: "user_name", name: "user[name]", type: "text", value: "value"]) |> Phoenix.HTML.safe_to_string()
"<input id=\"user_name\" name=\"user[name]\" type=\"text\" value=\"value\">"

We can reimplement the tag widget as a macro that expands into a complete {:undead_eex, ...} template, which will be embedded in a larger template and ultimately flattened into the larger template (for further optimization). For example, let’s try to relicate the above:

iex> import PhoenixUndeadView.Widgets
PhoenixUndeadView.Widgets
iex> Macro.expand(quote(do: tag(:input, [id: "user_name", name: "user[name]", type: "text", value: "value"])), __ENV__)
{:undead_eex,
 [
   static: "<input id=\"user_name\" name=\"user[name]\" type=\"text\" value=\"value\">"
 ]}

The output is pretty much the same (except that it is properly tagged as a compiled template should be). But values in input tags are often dynamic. So let’s make it dynamic:

iex> Macro.expand(quote(do: tag(:input, [id: "user_name", name: "user[name]", type: "text", value: @user.name])), __ENV__)
{:undead_eex,
 [
   static: "<input id=\"user_name\" name=\"user[name]\" type=\"text\" value=\"",
   dynamic: {:html_escape,
    [context: PhoenixUndeadView.Widgets, import: Phoenix.HTML],
    [
      {{:., [],
        [
          {:@, [context: Elixir, import: Kernel],
           [{:user, [context: Elixir], Elixir}]},
          :name
        ]}, [], []}
    ]},
   static: "\">"
 ]}

We get a much more interesting compiled template, which follows the main design rules of PhoenixUndeadView

  1. What is static is always static
  2. What is dynamic is always dynamic

The above would be transformed so that @user.name becomes something like assigns[:user][:name] or something like that, but the UndeadEngine already handles that part.

The fact that tag is now a macro that receives literal AST terms allows me to split the attribute list into static and dynamic parts and optimize it by merging all static parts together. This is a simple but very high-yield optimization.

If this macro is expanded inside a larger template, the static part at the beginning and the end can be merged with other binaries, thus increasing efficiency.

There might still be some problems with variable scope , but I’m pretty happy with the general idea. I think there shuoldn’t be any actual problems, but I haven’t tested this properly yet in the “real world”.

Things might get even more succinct with sigils

I have reimplemented the tag template by creating the tuples above “by hand”.
If we define a ~U"" sigil (from undead) which returns a compiled template, we might define widgets in a more natural way by making them macros that return the sigil.

That’s not something I’d do with the tag macro because it needs to support a variable number of attributes, but it might make sense in other cases where everything has a fixed number of arguments.

Plans for the future

I’ll reimplement most widgets in Phoenix.HTML as macros and reimplement the engine so that it can make use of these templates.

I still haven’t found a good way to reimplement form_for, one can get good results with something like:

<% form = FormData.to_form(@changeset, options) %>
<%= form_tag "/url", method: "post" do %>
  <%= text_input(form, :name) %>
  <%= text_input(form, :surname) %>
<% end %>

where form_tag/3 and text_input/2 are macros. It’s possible that even reimplementing form_tag/3 might be too difficult in a sane way (probably not, Ecto does much crazier things with macro than what I’m doing here). In that case we can go back to my dirty proposal of having open_form() and close_form() macros which act as ordinary segments.

3 Likes

After the silence, some news!

(@josevalim, there are almost the news you’ve been waiting for regarding expansion of macros)

I can finally expand Elixir macros inline inside the templates and optimize the resulting template by merging the static binaries together. I still can’t compile my new templates into executable Elixir code but it doesn’t pose any hard problems.

I’ve yet again changed the template format and the nomenclature. Templates are now composed of segments. A segment is either static text or a quoted expression represented by <%= ... %> (dynamic segment), <% ... %> (dynamic segment with no output) or <%/ ... > (fixed segment).

Segments are now represented as:

{segment_tag, {contents, metadata}}

As you can see, templates and parts of templates are represented by a 2-tuple nested inside another 2-tuple. The slight change from the previous post is because it’s useful to be able to store metadata in the segments for error reporting. Why nested 2-tuples instead of a 3-tuple? It’s because 2-tuples are represented as themselves in Elixir’s AST, unlike 3-tuples which have a special meaning. For example, the form {atom, metadata, arg} when not is_list(arg) is not valid Elixir AST! so it’s better to avoid 3-tuples and more complex expressions and working with nested 2-tuples. @OvermindDL1 has refereed to this as “escaping Elixir’s AST” and I really like the expression.

So, back to the point. Working with the nested 2-tuples is hard and the format is quite artificial, so I have combinators that make it easier to build them (and in the future maybe even pattern match on them). I’d love to use records (from the excellent Record module, which BTW should be more well known) instead, but they havd the complication that they wouldn’t compile to nested 2-tuples…

So, what’s so great about using 2-tuples exactly? It’s the fact that the intermediate representation of the compiled templates is a valid Elixir quoted expression! It’s not executable Elixir code, of course (it requires a couple transformation steps to become executable Elixir code), but is something which I can feed into Macro.prewalk/2, which makes it trivial to traverse the expression without having to manually implement a tree traversal that respects my templates’ semantics. My templates’ semantics are now the semantics of normal Elixir code.

Template widgets as macros

Last post I’ve talked about the possibility to implement reusable widgets as macros which expand into undead templates. An undead template is a value of the form:

{UndeadEngine.Segment.UndeadTemplate, {segments, meta}}

(Remember that UndeadEngine.Segment.UndeadTemplate is just an atom name like :undead_template. The advantagte of using a more verbose name like the one above is that it reduces the chance of accidental name collisions. This should be made more hygienic in the future anyway, but for now it’s good enough)

Any macro that expands into nested 2-tuples like the above can be optimized by flattening the segments and merging the static parts. As a (quite functional) proof of concept, I’ve implemented the tag/2 macro (you can find the implementation here). It works like this:

iex(2)> tag(:input, [name: "user[name]", name: "user_name", value: ""])
{UndeadEngine.Segment.UndeadTemplate,
 {[
    {UndeadEngine.Segment.Static, {"<", []}},
    {UndeadEngine.Segment.Static, {"input", []}},
    {UndeadEngine.Segment.Static, {" ", []}},
    {UndeadEngine.Segment.Static, {"name=\"user[name]\"", []}},
    {UndeadEngine.Segment.Static, {" ", []}},
    {UndeadEngine.Segment.Static, {"name=\"user_name\"", []}},
    {UndeadEngine.Segment.Static, {" ", []}},
    {UndeadEngine.Segment.Static, {"value=\"\"", []}},
    {UndeadEngine.Segment.Static, {">", []}}
  ], []}}

The argument list given to tag/2 is static, so it will generate a series of static segments. On itself, this is not very impressive.

On the other hand, this is very impressive:

iex(3)> tag(:input, [name: "user[name]", name: "user_name", value: ""]) |> Optimizer.optimize(__ENV__)
{UndeadEngine.Segment.UndeadTemplate,
 {[
    {UndeadEngine.Segment.Static,
     {"<input name=\"user[name]\" name=\"user_name\" value=\"\">", []}}
  ], []}}

The optimizer has just recognized the template as purely static, and has just merged the static parts together. Our reusable widget has been compiled into the most efficient format possible. Now let’s make it harder. Usually, an HTML widget won’t be purely static. It will contain dynamic parts. For example:

iex(4)> Macro.expand(quote(do: tag(:input, [name: "user[name]", name: "user_name", value: value])), __ENV__) |> Optimizer.optimize(__ENV__)
{UndeadEngine.Segment.UndeadTemplate,
 {[
    {UndeadEngine.Segment.Static,
     {"<input name=\"user[name]\" name=\"user_name\" value=\"", []}},
    {UndeadEngine.Segment.Dynamic,
     {{{:., [], [PhoenixUndeadView.Template.HTML, :html_escape]}, [],
       [{:value, [], Elixir}]}, []}},
    {UndeadEngine.Segment.Static, {"\">", []}}
  ], []}}

The system has detected that parts of the template are static and other parts are dynamic, and the optimizer has compiled the template into three segments: a static segment, a dynamic segment and a static segment. Again we see that the widget has been as optimized as possible.

Now let’s try it with a real template:

<% a = 2 %>
Blah blah blah

<%= tag(:input, [name: "user[name]", id: "user_name", value: @user.name]) %>

<%= a %>

Blah blah

It’s intuitively obvious that the template contains some dynamic parts and some static parts. It also contains the tag/2 macro which has an output that’s mostly static. Only the value attribute is dynamic. When we compile it, we get the following raw output (converted into Elixir code - not a quoted expression! Remember, we can do it using Macro.to_string() because the templates are valid AST):

{UndeadEngine.Segment.UndeadTemplate,
 {[
    {UndeadEngine.Segment.DynamicNoOutput, {a = 2, [line: 1]}},
    {UndeadEngine.Segment.Static, {"\nBlah blah blah\n\n", []}},
    {UndeadEngine.Segment.Dynamic,
     {tag(:input,
        name: "user[name]",
        id: "user_name",
        value: __MODULE__.fetch_assign(var!(assigns), :user).name()
      ), [line: 4]}},
    {UndeadEngine.Segment.Static, {"\n\n", []}},
    {UndeadEngine.Segment.Dynamic, {a, [line: 6]}},
    {UndeadEngine.Segment.Static, {"\n\nBlah blah\n", []}}
  ], []}}

Although the expression is quite complex, if you look carefully you can recognize the static and dynamic parts in the template above. You can also see that the tag/2 macro hasn’t been expanded yet. If we expand the macro and optimize it, we get the following:

{UndeadEngine.Segment.UndeadTemplate,
 {[
    {UndeadEngine.Segment.DynamicNoOutput, {a = 2, [line: 1]}},
    {UndeadEngine.Segment.Static,
     {"\nBlah blah blah\n\n<input name=\"user[name]\" id=\"user_name\" value=\"", []}},
    {UndeadEngine.Segment.Dynamic,
     {PhoenixUndeadView.Template.HTML.html_escape(Fixtures.fetch_assign(assigns, :user).name()),
      []}},
    {UndeadEngine.Segment.Static, {"\">\n\n", []}},
    {UndeadEngine.Segment.Dynamic, {a, [line: 6]}},
    {UndeadEngine.Segment.Static, {"\n\nBlah blah\n", []}}
  ], []}}

As you can see, the static parts of the tag at the beginning and at the end have been merged into the static parts before and after the tag, so minimize the number of segments.

As long as the macros expand into the appropriate format, thee optimizations are always possible.

What’s missing

I’ve taken a big detour with the goal of being able to expand macros inside the templates and optimize their result. I now have a very general framework to create optimized widgets, and it’s easy for other users to create their own libraries of optimized widgets using some basic combinators.

What is now missing is a way of compiling these templates into Elixir code that actually generates the iolist with the rendered template. The implementation doesn’t pose any particular challenges, and I’ll get to it soon. As you probably remember from prior posts, older versions of this project already had the capability to compile into quite efficient iolists, it’s just that the internal format of the templates has changed so much that I’ve had to scrap the old compiler and must reimplement a new one.

The basic idea is simple: you just need to compile the segments tagged as UndeadEngine.Segment.UndeaedTemplate into blocks that run their expressions in order, assign them to variables and return them as a list. It will be even simpler than what I was doing in previous versions because before I was actually compiling the templates into Elixir AST and then parsing that AST again, just to finally compile it into AST… I can be much more efficient now.

Source code

The code is, as always, on Github: https://github.com/tmbb/phoenix_undead_view

It’s still a little rough and some parts need to be refactored. The meat of the project is in the Tag module, which implements the “self-optimizing” tag/2 macro and the Optimizer module, which expands the macros in the template and optimizes it as much as possible by flattening it and merging the static segments together.

EDIT: Even if it turns out this architecture is not a good fit for something like LiveView, I have shown that with the thelp of macros such as tag/2 and with the optimization steps I can produce much better code than the default Phoenix templates and their functions-based widgets which must rerender everything (even some parts tat are actually static) each time the template is called. As I’ve said before, I believe the Phoenix engine could take some ideas from this implementation.

4 Likes

Sure you can, a binding like {:blah, [], Elixir} matches that pattern. :slight_smile:

Looking good though!

1 Like

Ok, arg can be an atom too, but it can’t be a general Elixir value, which I need for this to work.

Compiling templates into quoted expressions

(again, all examples are pretty simple, but this should work on arbitrarily complex templates)

As always, you can lookup the example (yes, singular, it’s getting late here!) on the github repo, but for those who are short on time, I reproduce them here. I’ve converted the quoted expressions into Elixir code because they are easier to read that way. When we write the quoted expressions into text, variable hygiene is lost, but rest assured that name collisions won’t happen.

Example template:

<% a = 2 %>
Blah blah blah

<%= tag(:input, [name: "user[name]", id: "user_name", value: @user.name]) %>

<%= a %>

Blah blah

Full template:

a = 2

tmp_3 =
  case(PhoenixUndeadView.Template.HTML.html_escape(fetch_assign(assigns, :user).name())) do
    {:safe, data} ->
      data

    bin when is_binary(bin) ->
      Plug.HTML.html_escape_to_iodata(bin)

    other ->
      Phoenix.HTML.Safe.to_iodata(other)
  end

tmp_5 =
  case(a) do
    {:safe, data} ->
      data

    bin when is_binary(bin) ->
      Plug.HTML.html_escape_to_iodata(bin)

    other ->
      Phoenix.HTML.Safe.to_iodata(other)
  end

{:safe,
 [
   "\nBlah blah blah\n\n<input name=\"user[name]\" id=\"user_name\" value=\"",
   tmp_3,
   "\">\n\n",
   tmp_5,
   "\n\nBlah blah\n"
 ]}

Static part:

{:safe,
 [
   "\nBlah blah blah\n\n<input name=\"user[name]\" id=\"user_name\" value=\"",
   "\">\n\n",
   "\n\nBlah blah\n"
 ]}

Dynamic part:

a = 2

tmp_3 =
  case(PhoenixUndeadView.Template.HTML.html_escape(fetch_assign(assigns, :user).name())) do
    {:safe, data} ->
      data

    bin when is_binary(bin) ->
      Plug.HTML.html_escape_to_iodata(bin)

    other ->
      Phoenix.HTML.Safe.to_iodata(other)
  end

tmp_5 =
  case(a) do
    {:safe, data} ->
      data

    bin when is_binary(bin) ->
      Plug.HTML.html_escape_to_iodata(bin)

    other ->
      Phoenix.HTML.Safe.to_iodata(other)
  end

{:safe, [tmp_3, tmp_5]}

Plans for the future

Documentation

I need to write a design document to explain the basic ideas behind the undead compiler and to discuss the tradeoffs of certain implementation choices

Features

The engine should now have feature parity with the default Phoenix engine. I don’t know if I’m going to add more features than what it already has.

Widgets

For this to be useful, I need to write a library of reusable macro widgets. The tag/2 macro is a good combinator, on top of which I can probably implement almost everything in the phoenix_html package.

Performance

This engine should be even more efficient than the default phoenix engine as long as reusable widgets are implemented as macros instead of functions (like my tag/2 macro above).

Transport protocol

I have to implement a transport protocol so that the dynamic parts can be set to the browser (and write the necessary Elixir and Javascript code for it to work, of course). I think I’ll need help on this one.

2 Likes

PS: I need help with the frontend parts not because I can’t do it (I can, it’s not that hard), bur because I’m not sure I can’t make it as efficient as possible andI’m not sure of what the best API would be.

The basic idea is to render something like:

<span data-undead-id="undead-id" data-undead-channel>
  ... (this is the initial html)
</span>

<script type="application/json" undead-widget-id="undead-id">
  [
     "Static#1",
     "Static#2",
     ... // literal JSON
  ]
</script>

Then, the javascript at the end of the page can join the appropriate channels. Upon receiving a message from the server, the javascript will decode the message into a list of strings and intersperse them into the static parts above. This can be done for many widgets, not only a single widget.

Then it can use morphdom or a similar library to merge the new DOM into the old one.

Where to go from now

There are two main avenues to explore now:

  1. Incorporate the template optimizations into “normal” Phoenix templates and develop a library of macro widgets so that more optimizations are possible

  2. Push forward into implementing my “full” copy of PhoenixLiveView, only with the unholy power of human sacrifice and the blood of virgins

1. Incorporating the improvements into “normal” (static) Phoenix templates

These improvements only make sense if I reimplement most widgets in Phoenix.HTML as macros. Otherwise, it’s probably not worth it… While I generate “cleaner” code (from the perspective of a human reader) than the default Phoenix templates, it’s not any faster, and the code complexity is much higher. The EEx engine used by Phoenix is dead-simple. My approach requires a custom EEx engine (actually simpler than the Phoenix one) and running a whole compiler on the output of said engine.

So if I decide to go this way, I will start by reimplementing the widgets in Phoenix.HTML as undead macros (I’ve just made up this term right now to mean "macros that compiler to an optimizable undead container). I think I can make it work in a backwards-compatible way (for some rather generous meaning of “backwar-compatible”).

The main problem with converting functions to macros is that macros don’t play well with higher order functions, unlike functions themselves, which are more composable. The only advantage of macros is that they can inspect the AST of their arguments and optimize their result at compile-time.

For normal static templates it’s not even clear that using my macros instead of functions makes that much of a difference. Theoretically it should, though, as it reduces the overhead of function calls and avoids calling functions for things that are actually static. The only way to be sure is to implement it and measuring performance.

2. Implement an end-to-end system ready to drop into a Phoenix app

If I decide to go this way, the best place to steal ideas from is probably Drab. Pinging @grych in case he wants to contribute with his experience.

Since I’m stealing from Drab, I’ll have something like Commanders (should I rename them to Necromancer or something to keep with the Undead theme?), which communicate bidirectionally with the browser. A Commander can only do two things:

  1. Receive events from the browser.
  2. Send deltas into the browser. A delta is a list of the strings that correspond to the parts that have changed. A client-side library like morphdom will make sense of the changes.

A Commander does not send arbitrary Javascript to the browser to be executed. This is just to keep the scope limited, it’s probably easy to make the server send Javascript to the browser.

In this example I’ll work with a commander named MyUndeadCommander. I’d like to render the initial HTML (and accompanying JSON) as:

<%= MyUndeadCommander.render() %>

It would generate the following parameters as random UUIDs:

  1. channel topic (for communication)
  2. widget id

The javascript at the end of the page would then join the appropriate channels. I’m still not sure how I want to match channels to commanders, but I’ll probably just copy what Drab does.

I’ll look at option number 1 first, because it seems simpler (and the way forward is clear). My second option requires lots of complex design decisions, and requires establishing conventions to communicate between the Javascript on the page and the Elixir code on the server.

2 Likes

I’ve found a way of compiling form(form_data, action, options, fun) into something sensible that optimizes well. I assume the fun is of the form fn arg -> body end. Because my form/4 is a macro, it receives a quoted expression from which I can extract both arg and body. The expression arg should be a variable (i.e. {name, meta, context} when is_atom(name) and is_list(meta) and is_atom(context)). Otherwise I raise an error (I could fallback to the normal form_for/4 instead, but an error seems better).

Then, I do something like:

{name, meta, _context} = arg
# create a safe hygienic variable
new_arg = {name, meta, __MODULE__}
# replace the old variable by the new hygienic variable
new_body = substitute(body, arg, old_arg)
# Do something like this, only more complex because I'm working with undead segments
# instead of literal Elixir AST:
quote do
  unquote(new_arg) = FormData.to_form(new_arg)
  unquote(new_body)
end

The new variable will leak outside the quoted expression, of course, but it’s now in a different context, so it shouldn’t be accessible to the rest of the template. The only way for a variable of the same name to be created if it is created by the PhoenixUndeadView.Template.Widgets.Formmodule, which doesn’t create any other variables.

Maybe it could become safer by adding a :counter to the metadata? I could then make sure the counter would be unique to the function that generates the AST above. That way, even if the PhoenixUndeadView.Template.Widgets.Form generates variables somewhere else, there will be no problems.

1 Like

Showtime: compiling forms

The template:

<% f = 2 %>
Blah blah blah

<%= form @changeset, action, [], fn f -> %>
  <%= text_input(f, :name) %>
  <%= text_input(f, :surname) %>
  <%= number_input(f, :age) %>
<% end %>

<%= f %>

Blah blah

The full template:

f = 2

tmp_3_dynamic =
  case(PhoenixUndeadView.Template.HTML.html_escape(action)) do
    {:safe, data} ->
      data

    bin when is_binary(bin) ->
      Plug.HTML.html_escape_to_iodata(bin)

    other ->
      Phoenix.HTML.Safe.to_iodata(other)
  end

tmp_5_fixed =
  case(Plug.CSRFProtection.get_csrf_token_for(action)) do
    {:safe, data} ->
      data

    bin when is_binary(bin) ->
      Plug.HTML.html_escape_to_iodata(bin)

    other ->
      Phoenix.HTML.Safe.to_iodata(other)
  end

f = Phoenix.HTML.FormData.to_form(fetch_assign(assigns, :changeset), [])

tmp_8_fixed =
  case(PhoenixUndeadView.Template.Widgets.Form.FormInputs.input_name_to_string(f)) do
    {:safe, data} ->
      data

    bin when is_binary(bin) ->
      Plug.HTML.html_escape_to_iodata(bin)

    other ->
      Phoenix.HTML.Safe.to_iodata(other)
  end

tmp_10_fixed =
  case(PhoenixUndeadView.Template.Widgets.Form.FormInputs.input_name_to_string(f)) do
    {:safe, data} ->
      data

    bin when is_binary(bin) ->
      Plug.HTML.html_escape_to_iodata(bin)

    other ->
      Phoenix.HTML.Safe.to_iodata(other)
  end

tmp_12_fixed =
  case(PhoenixUndeadView.Template.Widgets.Form.FormInputs.input_name_to_string(f)) do
    {:safe, data} ->
      data

    bin when is_binary(bin) ->
      Plug.HTML.html_escape_to_iodata(bin)

    other ->
      Phoenix.HTML.Safe.to_iodata(other)
  end

tmp_14_fixed =
  case(PhoenixUndeadView.Template.Widgets.Form.FormInputs.input_name_to_string(f)) do
    {:safe, data} ->
      data

    bin when is_binary(bin) ->
      Plug.HTML.html_escape_to_iodata(bin)

    other ->
      Phoenix.HTML.Safe.to_iodata(other)
  end

tmp_16_fixed =
  case(PhoenixUndeadView.Template.Widgets.Form.FormInputs.input_name_to_string(f)) do
    {:safe, data} ->
      data

    bin when is_binary(bin) ->
      Plug.HTML.html_escape_to_iodata(bin)

    other ->
      Phoenix.HTML.Safe.to_iodata(other)
  end

tmp_18_fixed =
  case(PhoenixUndeadView.Template.Widgets.Form.FormInputs.input_name_to_string(f)) do
    {:safe, data} ->
      data

    bin when is_binary(bin) ->
      Plug.HTML.html_escape_to_iodata(bin)

    other ->
      Phoenix.HTML.Safe.to_iodata(other)
  end

tmp_20_dynamic =
  case(f) do
    {:safe, data} ->
      data

    bin when is_binary(bin) ->
      Plug.HTML.html_escape_to_iodata(bin)

    other ->
      Phoenix.HTML.Safe.to_iodata(other)
  end

{:safe,
 [
   "\nBlah blah blah\n\n<form action=\"",
   tmp_3_dynamic,
   "\" accept_charset=\"UTF-8\"><input name=\"_csrf_token\" type=\"hidden\" value=\"",
   tmp_5_fixed,
   "\"><input name=\"_utf8\" hidden=\"hidden\" value=\"✓\">\n  <input name=\"",
   tmp_8_fixed,
   "[name]\" id=\"",
   tmp_10_fixed,
   "_name\" type=\"text\">\n  <input name=\"",
   tmp_12_fixed,
   "[surname]\" id=\"",
   tmp_14_fixed,
   "_surname\" type=\"text\">\n  <input name=\"",
   tmp_16_fixed,
   "[age]\" id=\"",
   tmp_18_fixed,
   "_age\" type=\"number\">\n</form>\n\n",
   tmp_20_dynamic,
   "\n\nBlah blah\n"
 ]}

The code above is long, but the most important part is the result:

{:safe,
 [
   "\nBlah blah blah\n\n<form action=\"",
   tmp_3_dynamic,
   "\" accept_charset=\"UTF-8\"><input name=\"_csrf_token\" type=\"hidden\" value=\"",
   tmp_5_fixed,
   "\"><input name=\"_utf8\" hidden=\"hidden\" value=\"✓\">\n  <input name=\"",
   tmp_8_fixed,
   "[name]\" id=\"",
   tmp_10_fixed,
   "_name\" type=\"text\">\n  <input name=\"",
   tmp_12_fixed,
   "[surname]\" id=\"",
   tmp_14_fixed,
   "_surname\" type=\"text\">\n  <input name=\"",
   tmp_16_fixed,
   "[age]\" id=\"",
   tmp_18_fixed,
   "_age\" type=\"number\">\n</form>\n\n",
   tmp_20_dynamic,
   "\n\nBlah blah\n"
 ]}

Most of the variables that appear in the final list are actually fixed and not dynamic (fixed variables end in _fixed and dynamic variables end in _dynamic), that is, they only need to be re-rendered when the data changes. This is a dangerous optimization, because it assumes the for name doesn’t change. This is true for sane templates, but not for “insane” ones. The problem is that there’s a tradeoff between how much you tag as dynamic and as much as you can optimize.

The line f = Phoenix.HTML.FormData.to_form(fetch_assign(assigns, :changeset), []) seems problematic, but it’s not as unsafe as it looks. The corresponding quoted expression is:

{:=, [],
    [
      {:f, [line: 4, counter: -576_460_752_303_422_203], PhoenixUndeadView.Template.Widgets.Form},
      {{:., [], [Phoenix.HTML.FormData, :to_form]}, [],
       [{:fetch_assign, [line: 4], [{:assigns, [line: 4, var: true], nil}, :changeset]}, []]}
    ]},

which means the variable is not accessible by the code written by the user (the user-defined variable f is {:f, [line: 5], nil}).

NOTES: The numbers in the variables are unique but not sequential because making them sequential would complicate the implementation too much and those variables are not user-facing anyway.

1 Like

Considering you can have it only support HTML and not something else like EMail or so (javascript DOM updating, etc…), then you can easily make a lot of assumptions there too. :slight_smile:

I’ve learned a lot the past few years, not certain if what I know is always the best but I did a lot of benchmarking when making bucklescript-tea too so I think my methods are at least the common and efficient ones if whatever is needed to help?

+1 Lol

Although merging it into Phoenix as it’s own EEx replacement engine for specific html work could be quite interesting…

That conceptually works rather well actually… ^.^

You will need to handle some javascript at the very least from the server, else interpolation in all ways becomes extremely difficult to handle well and the interface loses performance to the end-user.

1 Like

I can generate any binary. This engine doesn’t care about HTML, and I don’t intend to make it HTML specific.

BTW, if you look at the repo (and the example above) you can see how I’ve managed to implement form_for/4 in a sane way.

1 Like

This engine is already compatible with the default Phoenix engine. It doesn’t generate the same iolists, but it should return the same binary output (needs testing when I have the time for it, of course, but it’s written to be compatible.

All Phonix widgets can be used in this template, it’s just that they’re not optimized. To optimize the widget’s output (merge static parts together, etc), the widgets must be implemented as macros which can expand into something sensible depending on whether the AST given as argument is dynamic or static.

EDIT: which means that merging this into Phoenix is only worth it if the default widgets are implemented as macros.

Depending on how the user used them, they might be indistinguishable, but things won’t be backward comptible. You can’t use macros as arguments to higher order functions. For example: Enum.map(&macro/1, list) doesn’t really work like the user expects.

These comptibility breaks are probably a dealbreaker for it to be merged into Phoenix.

1 Like

I have to find out how easy it is to replace the default EEx engine in a phoenix application. I know it’s relatively simple to add new engines for new filetypes, but I wonder if it is easy to override the dafault one. If so, using Undead templates would be as simple as flipping a config option and importing the right widgets.

1 Like

Well I mean for the PhoenixUndeadView part, not just the EEx engine. :slight_smile:

Yeah it will require the redesign of phoenix_html by far.

Well drab did it by adding a new extension handler (`*.drab instead of *.eex), maybe you could do it by replacing an existing extension?

That’s not too hard. phoenix_html is not that big. It’s more docstrings than actual code. The problems is that the widgets will have to become macros, which isn’t 100% backward compatible, and I don’t even know if the performance impact is that

Actually, my (incomplete) version of phoenix_html is much bigger because I have to analyze a bunch of extra things at compile time.

EDIT: @OvermindDL1, you might find it interesting to look at the code inside the lib/phoenix_undead_view/template/widgets/ directory. It’s what I imagine Spirit’s source might look like (I don’t know enough C++ template language to take a look myself).

1 Like

You want to see some C++ template code? Take a look at:

That entire mass-of-horror-of-a-file is both the fastest uint parser in the world (same with basically all the other parsers in spirit) because it creates optimized assembly for every tiny little possible variant of unsigned integral parsing (you should see the real typ eparser!) that you could ever need, and that entire file I replicate (significantly less efficiently) in ExSpirit at:

C++ Templates are magic and wonderful, but their syntax is like a combination of Haskell, lambda calculus, and a bit of lovecraftian horror. Nothing you could do in Elixir would ever look as abhorrant. ^.^

What i’m doing is not abhorrant, it’s just that I’m obsessively traversing the AST of the arguments of the macro to separate the static frol the dynamic parts. I’s nothing especially bad, it’s just case analysis all the way down

1 Like

I’m now compiling the template into a DOM node and some JSON. The JSON contains the static parts, which will be merged with the dynamic parts sent by the server to build the DOM node. An example:

  <% a = 2 %>
  <% fixed = a %>
  Static part #1

  <%/ fixed %>

  <%= if a > 1 do %>
    <%= a %>
  <% else %>
    Nope
  <% end %>

  Static part #2

Which compiles to:

_ = assigns
element_id = Integer.to_string(:rand.uniform(4_294_967_296), 32)
a = 2
fixed = a

tmp_4_fixed =
  case(fixed) do
    {:safe, data} ->
      data

    bin when is_binary(bin) ->
      Plug.HTML.html_escape_to_iodata(bin)

    other ->
      Phoenix.HTML.Safe.to_iodata(other)
  end

tmp_6_dynamic =
  case(
    case(a > 1) do
      x when :erlang.orelse(:erlang."=:="(x, nil), :erlang."=:="(x, false)) ->
        {:safe, ["\n  Nope\n"]}

      _ ->
        tmp_2_2_dynamic =
          case(a) do
            {:safe, data} ->
              data

            bin when is_binary(bin) ->
              Plug.HTML.html_escape_to_iodata(bin)

            other ->
              Phoenix.HTML.Safe.to_iodata(other)
          end

        {:safe, ["\n  ", tmp_2_2_dynamic, "\n"]}
    end
  ) do
    {:safe, data} ->
      data

    bin when is_binary(bin) ->
      Plug.HTML.html_escape_to_iodata(bin)

    other ->
      Phoenix.HTML.Safe.to_iodata(other)
  end

{:safe,
 [
   "<span id=\"",
   element_id,
   "\">\n\nStatic part #1\n\n",
   tmp_4_fixed,
   "\n\n",
   tmp_6_dynamic,
   "\n\nStatic part #2\n</span>\n\n<script type=\"application/json\" undead-id=\"",
   element_id,
   "\">\n[\n  \"\",\n  \"\\n\\nStatic part #1\\n\\n",
   Phoenix.HTML.escape_javascript(tmp_4_fixed),
   "\\n\\n\",\n  \"\\n\\nStatic part #2\\n\"\n]\n</script>"
 ]}

The expression above is complex, but it’s not that hard to follow. It’s a span tag containing a DOM node (actually plaintext, but that shouldn’t be a problem for something like morphdom) followed by a script tag that contains some JSON.

Note that:

  1. I generate a unique ID for the DOM node. Maybe it should be possible to specify the element ID in the assigns?
  2. I start with _ = assigns so that writing a template that doesn’t use the assigns doesn’t raise a warning.

The above will render into the following (the assigns don’t matter because it’s independent from the assigns):

<span id="1IE7FC5">

Static part #1

2


  2


Static part #2
</span>

<script type="application/json" undead-id="1IE7FC5">
[
  "",
  "\n\nStatic part #1\n\n2\n\n",
  "\n\nStatic part #2\n"
]
</script>

The code that does the above is already available, but it’s a mess. I’ll refactor it (and delete dead code) ASAP.

2 Likes

It looks like it should be easy to ug this into a Drab commander (probably a shared commander). Drab already does a lot of yhe grunt work of bridging the world of HTTP requests and websockets, so it’s probably easy to use to handle the undead templates.

@grych might be interested in the last post in this series.

1 Like

As I’ve said before, these templates should be comparible with Phoenix (only more optimized). I’d like to benchmark compilation time and runtime performance of these templates agains the default engine. But I really don’t know what the relevant metrics are. The time it takes to render the template into an iolist is the obvious one. But I wonder if there is a way to measure how long it takes to send the bytes over a socket, and whether it is relevant.

Ultimately, I’d like to integrate this with Phoenix (the static part only). Is this something the Phoenix team would be ok with? It’s probably only worth it if it’s faster and if some HTML function helpers in phoenix_html are replaced by mostly backward compatible macros. Otherwise I can’t statically expand the widgets at compile time and make use of main optimization, which is to merge adjacent static segments together. If the phoenix team doesn’t agree with these changes, I’ll probably release the static part as a separate project.

Pinging @josevalim because he shown interest in these optimizations. By the way, I’ve solved allnproblems related the optimization of macro expansion. Everything “just works” now in an intuitive way.

A macro that can be optimized is just something which expands into a Segments.undead_container(segments, meta). This is itself a macro that hides the weird structure of the building blocks of my templates, which are nested 2-tuples (for technical reasons).

So it’s now actually very easy to write an HTML helper that optimizes properly: it’s just a macro that expands into the above (possibly using other helpers).

2 Likes