Exadra37
Implementing custom Markdown parser with the MD Library
I am trying to use this library to parse my custom markdown where I want to find calls to HEEX and then handle them but cannot figure out how to make the custom parser to be invoked.
The markdown:
## TEST
Some text before a card.
<.card image_path="/images/awesome.svg">Some nice card with an image on the left.</.card>
Continuing after the card.
My parser:
defmodule MasWeb.MdParser do
use Md.Parser
alias Md.Parser.Syntax.Void
@default_syntax Map.put(Void.syntax(), :settings, Void.settings())
@syntax @default_syntax
@impl true
def parse(input, state) do
# copied from the Md.Parser source code:
%State{ast: ast, path: []} = state = do_parse(input, state)
{"", %State{state | ast: Enum.reverse(ast)}}
end
end
The docs for Md.Parser say this:
Custom parsers might be used in syntax declaration when the generic functionality
is not enough.Let’s consider one needs a specific handling of links with titles.
The generic engine does not support it, so one would need to implement a custom parser
and instructMd.Parserto use it with:# config/prod.exs config :md, syntax: %{ custom: %{ {"![", MyApp.Parsers.Img}, ... } }Once the original parser would meet the
"!["binary, it’d callMyApp.Parsers.Img.parse/2.
The latter must proceed until the tag is closed and return the remainder and the updated state
as a tuple.
Adding the configuration to config/prod.exs doesn’t seem to make sense to me, thus I added it to config.exs :
config :md, syntax: %{
custom: [
{"<.", MasWeb.MdParser},
]
}
But then I get this error:
Erlang/OTP 25 [erts-13.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit:ns]
ERROR! the application :md has a different value set for key :syntax during runtime compared to compile time. Since this application environment entry was marked as compile time, this difference can lead to different behaviour than expected:
* Compile time value was not set
* Runtime value was set to: %{custom: [{"<.", MasWeb.MdParser}]}
To fix this error, you might:
* Make the runtime value match the compile time one
* Recompile your project. If the misconfigured application is a dependency, you may need to run "mix deps.compile md --force"
* Alternatively, you can disable this check. If you are using releases, you can set :validate_compile_env to false in your release configuration. If you are using Mix to start your system, you can pass the --no-validate-compile-env flag
10:57:41.583 [error] Task #PID<0.252.0> started from #PID<0.107.0> terminating
** (stop) "aborting boot"
(elixir 1.14.3) Config.Provider.boot/2
Function: &:erlang.apply/2
Args: [#Function<1.104735216/1 in Mix.Tasks.Compile.All.load_apps/3>, [md: "/home/developer/workspace/_build/dev/lib"]]
** (EXIT from #PID<0.107.0>) an exception was raised:
** (ErlangError) Erlang error: "aborting boot"
(elixir 1.14.3) Config.Provider.boot/2
If I try to recompile it as suggested in the output:
mix deps.compile md 1 ↵
==> md
Compiling 13 files (.ex)
== Compilation error in file lib/md/parser/default.ex ==
** (FunctionClauseError) no function clause matching in :erl_eval."-inside-an-interpreted-fun-"/1
The following arguments were given to :erl_eval."-inside-an-interpreted-fun-"/1:
# 1
{"<.", MasWeb.MdParser}
(stdlib 4.3) :erl_eval."-inside-an-interpreted-fun-"/1
(stdlib 4.3) erl_eval.erl:898: :erl_eval.eval_fun/8
/home/developer/workspace/deps/md/lib/md/parser/default.ex:1: (file)
/home/developer/workspace/deps/md/lib/md/parser/default.ex:1: (file)
(stdlib 4.3) erl_eval.erl:748: :erl_eval.do_apply/7
(stdlib 4.3) erl_eval.erl:136: :erl_eval.exprs/6
/home/developer/workspace/deps/md/lib/md/parser/default.ex:1: Md.Engine.__before_compile__/1
could not compile dependency :md, "mix compile" failed. Errors may have been logged above. You can recompile this dependency with "mix deps.compile md", update it with "mix deps.update md" or clean it with "mix deps.clean md"
I have no idea how to recover form this error, thus I commented out the MD entry from my config.exs and tried instead to configure it through my custom parser:
defmodule MasWeb.MdParser do
use Md.Parser
alias Md.Parser.Syntax.Void
@default_syntax Map.put(Void.syntax(), :settings, Void.settings())
@syntax @default_syntax |> Map.merge(%{
# I think I am doing something wrong here
common: {"<.", MasWeb.MdParser}, # example from Md.Parser
})
@impl true
def parse(input, state) do
IO.inspect(input, label: "MD PARSER INPUT")
IO.inspect(state, label: "MD PARSER STATE")
%State{ast: ast, path: []} = state = do_parse(input, state)
{"", %State{state | ast: Enum.reverse(ast)}}
end
end
I can now compile but I never see the IO.inspect output, thus my custom parser is not being invoked, which I kind of expected to happen, but i needed to give it a try.
Any guidance how to proceed to get a custom parser configured in my Phoenix app?
Marked As Solved
mudasobwa
Guilty.
The docs of md must be updated and extended widely.
In the first place, you’ve found a bug in the custom parsers implementation, thanks for that. Fixed in v0.9.7.
Second, you kinda mix up custom parsers and syntax. The very correct change in config.exs would now be handled properly, the correct syntax is:
import Config
config :md, syntax: %{
custom: [{"<.", {MasWeb.MdParser, %{}}}]
}
The custom parser is an implementation of Md.Parser behaviour, so copy-pasting from the library source wouldn’t help much. Instead, you are supposed to implement the whole parser. The very naïve implementation would look like this:
defmodule MasWeb.MdParser do
alias Md.Parser.State
@behaviour Md.Parser
@impl true
def parse(input, state) do
# <.card image_path="/images/awesome.svg">
# Some nice card with an image on the left.
# </.card>
# Continuing after the card.
[tag, rest] = String.split(input, " ", parts: 2)
[content, rest] = String.split(rest, "</.#{tag}>", parts: 2)
{rest, %State{state | ast: [content | state.ast]}}
end
end
Once we use <. as a tag, <. would not be passed to the handler itself. Hence we got card in the first split and the inner tag content in the second split. We end up with rest, which must be passed back to the “main” parser as a continuation, and content which you should translate to AST yourself (because it’s a custom parser.) Here we simply return back a text node.
Sidenote: custom name is essential, common would not work, it’s used as a tag type.
Thanks for giving the library a try, please, don’t hesitate to ask if anything. I’d love to make some progress with its docs and tests, but unfortunately, it serves our current needs and I am like, ok, later, then ![]()
Also Liked
mudasobwa
I was under impression that is the Elixir community standard.
Exadra37
Thank you very much for the options you gave me to think about. I will tak a look to them for sure.
Yesterday night I made it work and it seems that it matches the approach you suggest here:
My code:
defmodule MasWeb.MdParser.Heex do
alias Md.Parser.State
@behaviour Md.Parser
@impl true
def parse(input, state \\ %State{})
def parse(input, state) do
# @TODO handle the case for a tag without attributes:
# * <.whatever/>
# * <.whatever>content</.whatever>
[tag, rest] = String.split(input, " ", parts: 2)
[content, rest] = String.split(rest, "</.#{tag}>", parts: 2)
[attrs, content] = String.split(content, ">", parts: 2)
# parses markdown inside an HEEX tag to HTML and then rebuild the HEEX tag
# to allow for the Phoenix.LiveView.HTMLEngine to the parse it as a regular
# *.heex file, thus keeping all the niceties for LiveView tracking? (need to
# double check my assumption)
html = Md.Parser.generate(content)
html = "<.#{tag} #{attrs}>#{html}</.#{tag}>"
{rest, %State{state | ast: [html | state.ast]}}
end
end
The custom Phoenix Engine:
defmodule MasWeb.PhoenixEngine do
# @link Inspired by: https://github.com/boydm/phoenix_markdown/blob/ef7b5f76f339babec688021080a70708d9ddf1c1/lib/phoenix_markdown/engine.ex#L22
@moduledoc """
a single public function (compile) that Phoenix uses to compile incoming templates. You should not need to call it yourself.
"""
@behaviour Phoenix.Template.Engine
@doc """
Callback implementation for `Phoenix.Template.Engine.compile/2`
Precompiles the String file_path into a function defintion, using the EEx and Earmark engines
The compile function is typically called for by Phoenix's html engine and isn't something
you need to call your self.
### Parameters
* `path` path to the template being compiled
* `name` name of the template being compiled
"""
def compile(path, _name) do
options = [
engine: Phoenix.LiveView.TagEngine,
file: path,
line: 1,
caller: __ENV__,
source: "",
tag_handler: Phoenix.LiveView.HTMLEngine
]
path
|> File.read!()
|> Md.generate(Md.Parser.Default, format: :none)
|> EEx.compile_string(options)
end
end
The config.exs:
config :phoenix, :template_engines,
# will handle all markdown files that have an extension *.html.md, e,g. test.html.md
md: MasWeb.PhoenixEngine
config :md, syntax: %{
custom: [{"<.", {MasWeb.MdParser.Heex, %{}}}]
}
Add md extension to Phoenix live reload in config/dev.exs
config :mas, MasWeb.Endpoint,
live_reload: [
patterns: [
~r"priv/static/.*(js|css|png|jpeg|jpg|gif|svg)$",
~r"priv/gettext/.*(po)$",
~r"lib/mas_web/(controllers|live|components)/.*(ex|heex|md)$"
]
]
The markdown file test.html.md:
## TEST
<.horizontal_left_card class="bg-transparent" image_path="/images/svg/storyset_mobile-encryption-amico.svg">
One of the key challenges that mobile developers face when it comes to securing their apps and APIs is the ability to think like an attacker. This is because attackers approach mobile app and API security from a different perspective and mindset than developers. They usually combined several techniques and chain the weaknesses and vulnerabilities of the mobile apps and their APIS to succeed on their intents.
Developers are typically focused on building functionality and features that meet user requirements, while also ensuring that the app is performant and easy to use. While security is certainly an important consideration, it is often not the primary focus of developers, who may not have a deep understanding of the various security risks that their app or API may face, and if they do they may not be aware how creative hackers can be on combining such security risks to mount a successful attack.
</.horizontal_left_card>
<.horizontal_right_card class="w-full" image_path="/images/svg/storyset_hacker-bro.svg">
In contrast, attackers are motivated by different factors, such as financial gain, political, ideological or social motives, or simply the challenge of exploiting vulnerabilities in the mobile and their APIs. They approach mobile app security and API security from a different perspective, actively seeking out weaknesses and vulnerabilities that they can exploit for their own purposes or who they work for, that can be a criminal organization, a state or just a company trying to get ahead of their competitors.
To be able to effectively secure a mobile app or API, it is therefore important for developers to be able to think like an attacker. This requires a deep understanding of the various techniques and tools that attackers use to exploit vulnerabilities in mobile apps and APIs, as well as the ability to anticipate potential attack vectors and design security controls that can mitigate these risks.
</.horizontal_right_card>
<.cta_newsletter ></.cta_newsletter>
The result:
At the moment my Phoenix 1.7 app isn’t using LiveView but I plan to do so, therefore I need to wait until I can be sure that this also works properly with LiveView tracking.
Would this be something you would consider to add support for in your Lib? If yes I can make the PR.
Exadra37
Wow. I didn’t expect this first class support
![]()
Tonight I will try your latest changes and then I will provide some feedback.
Afterwards I will try to add at least a simple quickstart example to your home page in the docs.
Popular in Questions
Other popular topics
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #javascript
- #code-sync
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance










