Thanks so much
RobertDober
MarioFlach
I’ve tested with different inputs and Floki.raw_html/2 definitely works great (with basic typography tags, lists, code blocks) so far.
1
RobertDober
MarioFlach
I started experimenting with the AST. Pretty straight forward for what I want to do:
defmodule GitGud.Web.Markdown do
@moduledoc """
Conveniences for rendering Markdown.
"""
@doc """
Renders a Markdown formatted `content` to HTML.
"""
@spec markdown(binary | nil) :: binary | nil
def markdown(nil), do: nil
def markdown(content) do
case Earmark.as_ast(content) do
{:ok, ast, _warnings} ->
ast
|> transform_ast()
|> Floki.raw_html()
end
end
#
# Helpers
#
defp transform_ast(ast) do
ast
|> Enum.map(&transform_ast_node/1)
|> List.flatten()
end
defp transform_ast_node({tag, _attrs, _ast} = node) when tag in ["code"], do: node
defp transform_ast_node({tag, attrs, ast}) do
{tag, attrs, transform_ast(ast)}
end
defp transform_ast_node(content) when is_binary(content) do
content = Regex.replace(~r/:([a-z0-1\+]+):/, content, &emojify_short_name/2)
auto_link(content, Regex.scan(~r/#[0-9]+|@[a-zA-Z0-9_-]+|[a-f0-9]{7}/, content, return: :index))
end
defp emojify_short_name(match, short_name) do
if emoji = Exmoji.from_short_name(short_name),
do: Exmoji.EmojiChar.render(emoji),
else: match
end
defp auto_link(content, []), do: content
defp auto_link(content, indexes) do
{content, rest, _offset} =
Enum.reduce(List.flatten(indexes), {[], content, 0}, fn {idx, len}, {acc, rest, offset} ->
{head, rest} = String.split_at(rest, idx - offset)
{link, rest} =
case String.split_at(rest, len) do
{"#" <> number, rest} ->
{{"a", [], ["##{number}"]}, rest} # TODO
{"@" <> login, rest} ->
{{"a", [{"class", "has-text-black"}], ["@#{login}"]}, rest} # TODO
{hash, rest} ->
{{"a", [], [{"code", [{"class", "has-text-link"}], [hash]}]}, rest} # TODO
end
{acc ++ [head, link], rest, idx+len}
end)
List.flatten(content, [rest])
end
end
7
RobertDober
I am afraid of what will happen with comments, as I deliberately chose to diverge from Floki's decision here.
But I can change this, given the value to refeed into Floki.raw_html/1 might provide, so please keep the great information flow up.
0
MarioFlach
Using Floki.raw_html/1 is just a means to an end until Earmark provides it’s own AST → HTML function. I would not bother much about compatibility between the two. Perhaps if the only divergence is for comments it might be easier to just change that…
In my experimental implementation, parsing the text content of each node works fine but I’m wondering if this is the way to go.
Basically, each time I encounter an AST node text content (ignoring "code" tags) , I
- use a regex to replace
:emoji:to unicode emojis. - use a regex to match
#N(issue reference),@USER(user mention) andffffff(commit hash).
defp transform_ast_node(content) when is_binary(content) do
content = Regex.replace(~r/:([a-z0-1\+]+):/, content, &emojify_short_name/2)
auto_link(content, Regex.scan(~r/#[0-9]+|@[a-zA-Z0-9_-]+|[a-f0-9]{7}/, content, return: :index))
end
Now 1) is replacing text directly while 2) injects new nodes (links) into the AST.
I have more complex use-cases that will inject new nodes into the AST but sometime, I will require to mark/flag theses as “already parsed” for further processing. This is something that already happens with "code" tags. I just want further processing to skip theses parts because there are already in their “final-state”.
1
RobertDober
Hmm it might then be a good idea to apply the robustness principle if not too costly.
Maybe the transformer should accept also extended tuples in the AST, e.g.
{:code, [], children, _}
That would allow AST transformers to leave their annotations.
However it will fence us in concerning later extensions of the AST format, an alternative would be to allow only
{:code, [], children, {:meta, _}}
1
RobertDober
RELEASE NOTES for the latest Earmark Release 
version 1.4.1 2019/09/24
-
277 Expose an AST to HTML Transformer
While it should be faster to callto_ast|>transformit cannot be used instead ofas_htmlyet
as the API is not yet stable and some subtle differences in the output need to be addressed. -
278 Implementing better GFM Table support
Because of compatility issues we use a new optiongfm_tablesdefaulting tofalsefor this.
Using this optionEarmarkwill implement its own table extension + GFM tables at the same
time. -
279 Languages in code blocks were limited to alphanum names, thus excluding, e.g. C#
-
281 Urls in links were URL endoded, that is actually a bug
It is the markdown author’s responsability to url encode her urls, if she does so correctly
we double encoded the url before this fix. -
282 Always create a
<tbody>in tables
Although strictly speaking a<tbody>is only needed when there is a<thead>, semantic
HTML suggests the presence of<tbody>anyway.
2
RobertDober
Slight change in the API, after some reflection, how an internal extension might look like.
I prefer to keep the cool name meta: for myself 
So the acceptable ast would be extended by a map rather where the custom: key shall be reserved for 3rd party applications
{tag, atts, children, %{custom: ...}}
c.f. Allow metadata in the ast passed into the transformer · Issue #288 · pragdave/earmark · GitHub
0
RobertDober
benchmarking shows no speed difference whatsover, indicating that parsing is the dominant speed factor
0