Yes I did! I’ve been doing too much OCaml and got them mixed up, it is ,, like in C++ (and in C++ you can even override operator,, lol). 
Edited, it look good? Sorry for the mis-remembering, the comma is like the comma operator in C++, I need to remember it that way. ^.^
I’ve managed to write a pretty functional proof of concept that doesn’t require AST rewriting of complex quoted expressions. Code is here:
I can generate “full” templates as well as templated containing only the static and dynamic parts of the template, for easier diffing over the network. As a positive side-effect, I my template engines generate code which is much cleaner than the code generated by the engine in Phoenix.HTML, and it might even be slightly faster.
I should benchmark it one of these days.
EDIT: code here: phoenix_undead_view/README.md at master · tmbb/phoenix_undead_view · GitHub
Link to the relevant engine? I think you have a couple ones around, so I am not sure which one I should look at.
There are 4 engines. Only the following 3 are important (the other one is a relic which I keep around for reference purposes):
UndeadEngineFull - generates the full webpage/component, for the initial server-side rendering. This is the equivalent to what Phoenix.HTML.Engine does already (and as you can see, the code was mostly copied from there). This template generates the following code: quoted expressions; code. The result is rendered to a list of alternating static and dynamic elements (i.e. static, dynamic, static, dynamic, …, dynamic, static). The list always starts and ends with a binary (even if it i an empty one ""), to make it easier to render the full text from diffs, but this can be optimized away if you don’t need diffing (and normal Phoenix templates don’t need it).
UndeadEngineDynamicParts - This engine renders only the dynamic parts rendered by the above template. The goal would be to render only the parts that change, send them over the network, and join them with the static parts on the client.
UndeadEngineStaticParts - This engine renders only the static parts. The goal is to send these parts to the client only once so that the client knows how to merge the dynamic parts into the DOM. I think of this as a diff of the template (even though we are always diffing agains the static part, and not against the previous version)
A normal page render requires sending the output of the Full engine (as the literal DOM) and the StaticParts engine (compiled into a Javascript function that knows how to concat the dynamic parts it will receive later). Whenever the server wants to change the DOM on the client, it must run the DynamicParts engine (and only this one!) and send the list into the client, which will merge the dynamic parts into the textual HTML and merge it into the DOM.
If all you want is to compare my engine to Phoenix.HTML.Engine, you only need to look at the UndeadEngineFull and at its output. The output of the engine is a quoted expression, but you should format it into source code using Macro.to_string() and Code.format_string!() because the quoted expressions are quite unreadable, even for simple templates.
Now, my 3 engines contain some common functionality. In fact, only the handle_body() callback is different for each engine. To simplify things, I always use a UndeadEngine, which contains the implementations of the common callbacks. Because I didn’t want to define a lot of code in the __using__ macro, I’ve put the actual implementation of the callbacks in the UndeadEngineImpl module. While these layers of indirections make my project more complex, they’re kinda required if I want to work on 3 similar engines at the same time, but you don’t need any of these if you only want the Full engine.
I haven’t tested this engine on complex templates, so it’s possible you might find a way of breaking it. I’m not sure I’m scoping the inner variables correctly, but that’s very easy to fix by adding the :counter attribute to the variables, changing the variable names according to the nesting level (which is easy because I can keep track of the nesting level if I want) or by wrapping the inner structures in function calls so that variables don’t leak.
You might need to read the Utils and Merger modules, which build and optimize the final AST.
Maybe the code is too unfinished for you to look at, but I think it’s a good proof of concept even if it leaks some variables in highly nested templates.
EDIT: Problems with variable scoping should be solved now!
PS: if you want we can discuss this over private messages, but I think the discussion might be interesting for the community in general.
The main thing that’s missing now from the Undead Engines is a way to expand macros and extract the constant and dynamic parts. That sounds really hard, but it’s not as complex as it might seem…
If you look at the code generated by the engines, it follows a very simple pattern (in fact, the pattern is even a little more restricted than that):
{:safe,
(
var1 = ...
var2 = ...
var3 = ...
...
varn = ...
[var1, var2, ..., varn]
)}
If I can recognize this pattern at compile time, I can “lift” it into a lower level and add the new variables to the lower level. Because the format is generated by my own code, it’s trivial to convert this quoted expression into a more manageable format. I’d be rewriting the AST, but a very simple and well-defined AST fragment. This means that the Engines can’t expand every macro, but they can expand every macro that generates a quoted expression as if it were generated by the Engine, which is a useful subset.
There are possibilities that avoid AST rewriting, like choosing to expand macros inside the Engine, instead of expanding them after the template has been compiled. The problem with that approach is that it only works for “wel behaved” macros, which have a return value that only depends on their arguments and the env outside of the quoted expression. If they depend on variables defined inside the quoted expression, they won’t work. Since we want the semantics of these Engines to be exactly equal to the semantics of the Phoenix.HTML.Engine, I should probably support the general case.
Anyway, supporting macro expansions is not a big deal, because the macros will be expanded anyway when the quoted expression is compiled. This is just an optimization. I’m a little short on time to promise I’ll get all of this to work soon…
Great job!
I think we should only have the UndeadEngineFull and we can assume that it will always return something in the shape of {:safe, {:__block__, [], [..., ..., [static, dyamic, ...]}}. This way we can extract static and dynamic out whenever we feel like it.
About macro expansion, I need to put a bit more thought into that. 
Yes, that’s totally possible. Having the three engines was mostly an optimization. The UndeadEngineDynamicParts engine was mostly an optimization. Depending on how the BEAM represents static binaries in list having this version might save some memory and time copying the binaries into the list.
Because most of the time templates are mostly large static binaries with small dynamic parts, if you’re copying those binaries into a list you’re wasting a lot of memory.
But you’re more familiar with the BEAM internals than I am.
Of course. This pet project of mine has always been more about thinking than writing code. You need to understand pretty well what’s happening and where you eant to go before starting to type the code, or you might code your way into a dead end… When you think your way through, the actual code is often obvious.
Anyway, did you understand the basic idea in my explanation? The defails are a little bit involved, but the basics are simple:
“Compile” the text into an abstract list of variable assignments in a format convenient for further transformation, not as literal AST nodes. That’s what my code does currently, as you have probably noticed.
Convert the abstract assignments into a quoted expression with (possibly nested) concrete assignments (i.e. literal AST nodes).
Expand the macros
Convert the quoted expression into the standard abstract format. This is not that hard because the AST was generated by us.
Recursively lift expressions matching the {:safe, block} format from lower levels into the upper levels
Optimize the resultant abstract lists
Compile them into quoted expressions again
This is mostly a repetition of what I said earlier, but, splt into easy understandable steps. The inly tricky step is 4. This is easy to understand conceptually and easy to describe in pseudocode, but not that trivial to implement. Not that it requires anything more clever than tree manipulation, though.
Just to be clear, this is NOT what I’m doing. This is what I’d like to do.
Bonus (if you can still follow my ramblings):
One “problem” of expanding macros inside the template is that the interesting expressions are “trapped” inside case expressions:
case(expression_to_macroexpand) do
{:safe, ...} -> ...
...
end
These expressions are easy to find, but I’m planning on compiling into an intermediate form such as:
{:__dynamic_expression__, expression_to_macroexpand}
This is easier to analyze statically than a case statement. Then, when generating the final quoted expression, I can compile these tuples into proper case statements.
Yup, I do follow everything up. But since you are in experiment land, I would like to suggest for you to finally implement this all directly in the engine instead of doing traversals. Then you only need to look at expressions in the format of {:safe, {:__block__, _, _}} for further inlining.