After the silence, some news!
(@josevalim, there are almost the news you’ve been waiting for regarding expansion of macros)
I can finally expand Elixir macros inline inside the templates and optimize the resulting template by merging the static binaries together. I still can’t compile my new templates into executable Elixir code but it doesn’t pose any hard problems.
I’ve yet again changed the template format and the nomenclature. Templates are now composed of segments. A segment is either static text or a quoted expression represented by <%= ... %> (dynamic segment), <% ... %> (dynamic segment with no output) or <%/ ... > (fixed segment).
Segments are now represented as:
{segment_tag, {contents, metadata}}
As you can see, templates and parts of templates are represented by a 2-tuple nested inside another 2-tuple. The slight change from the previous post is because it’s useful to be able to store metadata in the segments for error reporting. Why nested 2-tuples instead of a 3-tuple? It’s because 2-tuples are represented as themselves in Elixir’s AST, unlike 3-tuples which have a special meaning. For example, the form {atom, metadata, arg} when not is_list(arg) is not valid Elixir AST! so it’s better to avoid 3-tuples and more complex expressions and working with nested 2-tuples. @OvermindDL1 has refereed to this as “escaping Elixir’s AST” and I really like the expression.
So, back to the point. Working with the nested 2-tuples is hard and the format is quite artificial, so I have combinators that make it easier to build them (and in the future maybe even pattern match on them). I’d love to use records (from the excellent Record module, which BTW should be more well known) instead, but they havd the complication that they wouldn’t compile to nested 2-tuples…
So, what’s so great about using 2-tuples exactly? It’s the fact that the intermediate representation of the compiled templates is a valid Elixir quoted expression! It’s not executable Elixir code, of course (it requires a couple transformation steps to become executable Elixir code), but is something which I can feed into Macro.prewalk/2, which makes it trivial to traverse the expression without having to manually implement a tree traversal that respects my templates’ semantics. My templates’ semantics are now the semantics of normal Elixir code.
Template widgets as macros
Last post I’ve talked about the possibility to implement reusable widgets as macros which expand into undead templates. An undead template is a value of the form:
{UndeadEngine.Segment.UndeadTemplate, {segments, meta}}
(Remember that UndeadEngine.Segment.UndeadTemplate is just an atom name like :undead_template. The advantagte of using a more verbose name like the one above is that it reduces the chance of accidental name collisions. This should be made more hygienic in the future anyway, but for now it’s good enough)
Any macro that expands into nested 2-tuples like the above can be optimized by flattening the segments and merging the static parts. As a (quite functional) proof of concept, I’ve implemented the tag/2 macro (you can find the implementation here). It works like this:
iex(2)> tag(:input, [name: "user[name]", name: "user_name", value: ""])
{UndeadEngine.Segment.UndeadTemplate,
{[
{UndeadEngine.Segment.Static, {"<", []}},
{UndeadEngine.Segment.Static, {"input", []}},
{UndeadEngine.Segment.Static, {" ", []}},
{UndeadEngine.Segment.Static, {"name=\"user[name]\"", []}},
{UndeadEngine.Segment.Static, {" ", []}},
{UndeadEngine.Segment.Static, {"name=\"user_name\"", []}},
{UndeadEngine.Segment.Static, {" ", []}},
{UndeadEngine.Segment.Static, {"value=\"\"", []}},
{UndeadEngine.Segment.Static, {">", []}}
], []}}
The argument list given to tag/2 is static, so it will generate a series of static segments. On itself, this is not very impressive.
On the other hand, this is very impressive:
iex(3)> tag(:input, [name: "user[name]", name: "user_name", value: ""]) |> Optimizer.optimize(__ENV__)
{UndeadEngine.Segment.UndeadTemplate,
{[
{UndeadEngine.Segment.Static,
{"<input name=\"user[name]\" name=\"user_name\" value=\"\">", []}}
], []}}
The optimizer has just recognized the template as purely static, and has just merged the static parts together. Our reusable widget has been compiled into the most efficient format possible. Now let’s make it harder. Usually, an HTML widget won’t be purely static. It will contain dynamic parts. For example:
iex(4)> Macro.expand(quote(do: tag(:input, [name: "user[name]", name: "user_name", value: value])), __ENV__) |> Optimizer.optimize(__ENV__)
{UndeadEngine.Segment.UndeadTemplate,
{[
{UndeadEngine.Segment.Static,
{"<input name=\"user[name]\" name=\"user_name\" value=\"", []}},
{UndeadEngine.Segment.Dynamic,
{{{:., [], [PhoenixUndeadView.Template.HTML, :html_escape]}, [],
[{:value, [], Elixir}]}, []}},
{UndeadEngine.Segment.Static, {"\">", []}}
], []}}
The system has detected that parts of the template are static and other parts are dynamic, and the optimizer has compiled the template into three segments: a static segment, a dynamic segment and a static segment. Again we see that the widget has been as optimized as possible.
Now let’s try it with a real template:
<% a = 2 %>
Blah blah blah
<%= tag(:input, [name: "user[name]", id: "user_name", value: @user.name]) %>
<%= a %>
Blah blah
It’s intuitively obvious that the template contains some dynamic parts and some static parts. It also contains the tag/2 macro which has an output that’s mostly static. Only the value attribute is dynamic. When we compile it, we get the following raw output (converted into Elixir code - not a quoted expression! Remember, we can do it using Macro.to_string() because the templates are valid AST):
{UndeadEngine.Segment.UndeadTemplate,
{[
{UndeadEngine.Segment.DynamicNoOutput, {a = 2, [line: 1]}},
{UndeadEngine.Segment.Static, {"\nBlah blah blah\n\n", []}},
{UndeadEngine.Segment.Dynamic,
{tag(:input,
name: "user[name]",
id: "user_name",
value: __MODULE__.fetch_assign(var!(assigns), :user).name()
), [line: 4]}},
{UndeadEngine.Segment.Static, {"\n\n", []}},
{UndeadEngine.Segment.Dynamic, {a, [line: 6]}},
{UndeadEngine.Segment.Static, {"\n\nBlah blah\n", []}}
], []}}
Although the expression is quite complex, if you look carefully you can recognize the static and dynamic parts in the template above. You can also see that the tag/2 macro hasn’t been expanded yet. If we expand the macro and optimize it, we get the following:
{UndeadEngine.Segment.UndeadTemplate,
{[
{UndeadEngine.Segment.DynamicNoOutput, {a = 2, [line: 1]}},
{UndeadEngine.Segment.Static,
{"\nBlah blah blah\n\n<input name=\"user[name]\" id=\"user_name\" value=\"", []}},
{UndeadEngine.Segment.Dynamic,
{PhoenixUndeadView.Template.HTML.html_escape(Fixtures.fetch_assign(assigns, :user).name()),
[]}},
{UndeadEngine.Segment.Static, {"\">\n\n", []}},
{UndeadEngine.Segment.Dynamic, {a, [line: 6]}},
{UndeadEngine.Segment.Static, {"\n\nBlah blah\n", []}}
], []}}
As you can see, the static parts of the tag at the beginning and at the end have been merged into the static parts before and after the tag, so minimize the number of segments.
As long as the macros expand into the appropriate format, thee optimizations are always possible.
What’s missing
I’ve taken a big detour with the goal of being able to expand macros inside the templates and optimize their result. I now have a very general framework to create optimized widgets, and it’s easy for other users to create their own libraries of optimized widgets using some basic combinators.
What is now missing is a way of compiling these templates into Elixir code that actually generates the iolist with the rendered template. The implementation doesn’t pose any particular challenges, and I’ll get to it soon. As you probably remember from prior posts, older versions of this project already had the capability to compile into quite efficient iolists, it’s just that the internal format of the templates has changed so much that I’ve had to scrap the old compiler and must reimplement a new one.
The basic idea is simple: you just need to compile the segments tagged as UndeadEngine.Segment.UndeaedTemplate into blocks that run their expressions in order, assign them to variables and return them as a list. It will be even simpler than what I was doing in previous versions because before I was actually compiling the templates into Elixir AST and then parsing that AST again, just to finally compile it into AST… I can be much more efficient now.
Source code
The code is, as always, on Github: GitHub - tmbb/phoenix_undead_view: EEx engines that compile Phoenix templates into static and dynamic parts for better diffing over the network · GitHub
It’s still a little rough and some parts need to be refactored. The meat of the project is in the Tag module, which implements the “self-optimizing” tag/2 macro and the Optimizer module, which expands the macros in the template and optimizes it as much as possible by flattening it and merging the static segments together.
EDIT: Even if it turns out this architecture is not a good fit for something like LiveView, I have shown that with the thelp of macros such as tag/2 and with the optimization steps I can produce much better code than the default Phoenix templates and their functions-based widgets which must rerender everything (even some parts tat are actually static) each time the template is called. As I’ve said before, I believe the Phoenix engine could take some ideas from this implementation.
