Exadra37

Exadra37

Implementing custom Markdown parser with the MD Library

I am trying to use this library to parse my custom markdown where I want to find calls to HEEX and then handle them but cannot figure out how to make the custom parser to be invoked.

The markdown:

## TEST

Some text before a card.

<.card image_path="/images/awesome.svg">Some nice card with an image on the left.</.card>

Continuing after the card.

My parser:

defmodule MasWeb.MdParser do

  use Md.Parser

  alias Md.Parser.Syntax.Void

  @default_syntax Map.put(Void.syntax(), :settings, Void.settings())
  @syntax @default_syntax

  @impl true
  def parse(input, state) do
    # copied from the Md.Parser source code:
    %State{ast: ast, path: []} = state = do_parse(input, state)
    {"", %State{state | ast: Enum.reverse(ast)}}
  end
end

The docs for Md.Parser say this:

Custom parsers might be used in syntax declaration when the generic functionality
is not enough.

Let’s consider one needs a specific handling of links with titles.

The generic engine does not support it, so one would need to implement a custom parser
and instruct Md.Parser to use it with:

# config/prod.exs

config :md, syntax: %{
  custom: %{
    {"![", MyApp.Parsers.Img},
    ...
  }
}

Once the original parser would meet the "![" binary, it’d call MyApp.Parsers.Img.parse/2.
The latter must proceed until the tag is closed and return the remainder and the updated state
as a tuple.

Adding the configuration to config/prod.exs doesn’t seem to make sense to me, thus I added it to config.exs :

config :md, syntax: %{
  custom: [
    {"<.", MasWeb.MdParser},
  ]
}

But then I get this error:

Erlang/OTP 25 [erts-13.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit:ns]

ERROR! the application :md has a different value set for key :syntax during runtime compared to compile time. Since this application environment entry was marked as compile time, this difference can lead to different behaviour than expected:

  * Compile time value was not set
  * Runtime value was set to: %{custom: [{"<.", MasWeb.MdParser}]}

To fix this error, you might:

  * Make the runtime value match the compile time one

  * Recompile your project. If the misconfigured application is a dependency, you may need to run "mix deps.compile md --force"

  * Alternatively, you can disable this check. If you are using releases, you can set :validate_compile_env to false in your release configuration. If you are using Mix to start your system, you can pass the --no-validate-compile-env flag



10:57:41.583 [error] Task #PID<0.252.0> started from #PID<0.107.0> terminating
** (stop) "aborting boot"
    (elixir 1.14.3) Config.Provider.boot/2
Function: &:erlang.apply/2
    Args: [#Function<1.104735216/1 in Mix.Tasks.Compile.All.load_apps/3>, [md: "/home/developer/workspace/_build/dev/lib"]]
** (EXIT from #PID<0.107.0>) an exception was raised:
    ** (ErlangError) Erlang error: "aborting boot"
        (elixir 1.14.3) Config.Provider.boot/2

If I try to recompile it as suggested in the output:

 mix deps.compile md                                                                                                                                                                                                       1 ↵
==> md
Compiling 13 files (.ex)

== Compilation error in file lib/md/parser/default.ex ==
** (FunctionClauseError) no function clause matching in :erl_eval."-inside-an-interpreted-fun-"/1    
    
    The following arguments were given to :erl_eval."-inside-an-interpreted-fun-"/1:
    
        # 1
        {"<.", MasWeb.MdParser}
    
    (stdlib 4.3) :erl_eval."-inside-an-interpreted-fun-"/1
    (stdlib 4.3) erl_eval.erl:898: :erl_eval.eval_fun/8
    /home/developer/workspace/deps/md/lib/md/parser/default.ex:1: (file)
    /home/developer/workspace/deps/md/lib/md/parser/default.ex:1: (file)
    (stdlib 4.3) erl_eval.erl:748: :erl_eval.do_apply/7
    (stdlib 4.3) erl_eval.erl:136: :erl_eval.exprs/6
    /home/developer/workspace/deps/md/lib/md/parser/default.ex:1: Md.Engine.__before_compile__/1
could not compile dependency :md, "mix compile" failed. Errors may have been logged above. You can recompile this dependency with "mix deps.compile md", update it with "mix deps.update md" or clean it with "mix deps.clean md"

I have no idea how to recover form this error, thus I commented out the MD entry from my config.exs and tried instead to configure it through my custom parser:

defmodule MasWeb.MdParser do

  use Md.Parser

  alias Md.Parser.Syntax.Void

  @default_syntax Map.put(Void.syntax(), :settings, Void.settings())
  @syntax @default_syntax |> Map.merge(%{
    # I think I am doing something wrong here
    common: {"<.", MasWeb.MdParser}, # example from Md.Parser
  })

  @impl true
  def parse(input, state) do
    IO.inspect(input, label: "MD PARSER INPUT")
    IO.inspect(state, label: "MD PARSER STATE")
    %State{ast: ast, path: []} = state = do_parse(input, state)
    {"", %State{state | ast: Enum.reverse(ast)}}
  end
end

I can now compile but I never see the IO.inspect output, thus my custom parser is not being invoked, which I kind of expected to happen, but i needed to give it a try.

Any guidance how to proceed to get a custom parser configured in my Phoenix app?

Marked As Solved

mudasobwa

mudasobwa

Creator of Cure

Guilty.

The docs of md must be updated and extended widely.

In the first place, you’ve found a bug in the custom parsers implementation, thanks for that. Fixed in v0.9.7.

Second, you kinda mix up custom parsers and syntax. The very correct change in config.exs would now be handled properly, the correct syntax is:

import Config

config :md, syntax: %{
  custom: [{"<.", {MasWeb.MdParser, %{}}}]
}

The custom parser is an implementation of Md.Parser behaviour, so copy-pasting from the library source wouldn’t help much. Instead, you are supposed to implement the whole parser. The very naïve implementation would look like this:

defmodule MasWeb.MdParser do
  alias Md.Parser.State

  @behaviour Md.Parser

  @impl true
  def parse(input, state) do
    # <.card image_path="/images/awesome.svg">
    #   Some nice card with an image on the left.
    # </.card>
    # Continuing after the card.

    [tag, rest] = String.split(input, " ", parts: 2)
    [content, rest] = String.split(rest, "</.#{tag}>", parts: 2)
    {rest, %State{state | ast: [content | state.ast]}}
  end
end

Once we use <. as a tag, <. would not be passed to the handler itself. Hence we got card in the first split and the inner tag content in the second split. We end up with rest, which must be passed back to the “main” parser as a continuation, and content which you should translate to AST yourself (because it’s a custom parser.) Here we simply return back a text node.

Sidenote: custom name is essential, common would not work, it’s used as a tag type.


Thanks for giving the library a try, please, don’t hesitate to ask if anything. I’d love to make some progress with its docs and tests, but unfortunately, it serves our current needs and I am like, ok, later, then :slight_smile:

Also Liked

mudasobwa

mudasobwa

Creator of Cure

I was under impression that is the Elixir community standard.

Exadra37

Exadra37

Thank you very much for the options you gave me to think about. I will tak a look to them for sure.

Yesterday night I made it work and it seems that it matches the approach you suggest here:

My code:

defmodule MasWeb.MdParser.Heex do

  alias Md.Parser.State

  @behaviour Md.Parser

  @impl true
  def parse(input, state \\ %State{})
  def parse(input, state) do
    # @TODO handle the case for a tag without attributes:
    #       * <.whatever/>
    #       * <.whatever>content</.whatever>
    [tag, rest] = String.split(input, " ", parts: 2)
    [content, rest] = String.split(rest, "</.#{tag}>", parts: 2)
    [attrs, content] = String.split(content, ">", parts: 2)

    # parses markdown inside an HEEX tag to HTML and then rebuild the HEEX tag
    # to allow for the Phoenix.LiveView.HTMLEngine to the parse it as a regular 
    # *.heex file, thus keeping all the niceties for LiveView tracking? (need to
    # double check my assumption)
    html = Md.Parser.generate(content)
    html = "<.#{tag} #{attrs}>#{html}</.#{tag}>"

    {rest, %State{state | ast: [html | state.ast]}}
  end
end

The custom Phoenix Engine:

defmodule MasWeb.PhoenixEngine do

   # @link Inspired by: https://github.com/boydm/phoenix_markdown/blob/ef7b5f76f339babec688021080a70708d9ddf1c1/lib/phoenix_markdown/engine.ex#L22

  @moduledoc """
  a single public function (compile) that Phoenix uses to compile incoming templates. You should not need to call it yourself.
  """

  @behaviour Phoenix.Template.Engine

  @doc """
  Callback implementation for `Phoenix.Template.Engine.compile/2`

  Precompiles the String file_path into a function defintion, using the EEx and Earmark engines

  The compile function is typically called for by Phoenix's html engine and isn't something
  you need to call your self.

  ### Parameters
    * `path` path to the template being compiled
    * `name` name of the template being compiled

  """
  def compile(path, _name) do

    options = [
      engine: Phoenix.LiveView.TagEngine,
      file: path,
      line: 1,
      caller: __ENV__,
      source: "",
      tag_handler: Phoenix.LiveView.HTMLEngine
    ]

    path
    |> File.read!()
    |> Md.generate(Md.Parser.Default, format: :none)
    |> EEx.compile_string(options)
  end
end

The config.exs:

config :phoenix, :template_engines,
  # will handle all markdown files that have an extension *.html.md, e,g. test.html.md
  md: MasWeb.PhoenixEngine

config :md, syntax: %{
  custom: [{"<.", {MasWeb.MdParser.Heex, %{}}}]
}

Add md extension to Phoenix live reload in config/dev.exs

config :mas, MasWeb.Endpoint,
live_reload: [
  patterns: [
    ~r"priv/static/.*(js|css|png|jpeg|jpg|gif|svg)$",
    ~r"priv/gettext/.*(po)$",
    ~r"lib/mas_web/(controllers|live|components)/.*(ex|heex|md)$"
  ]
]

The markdown file test.html.md:

## TEST

<.horizontal_left_card class="bg-transparent" image_path="/images/svg/storyset_mobile-encryption-amico.svg">
One of the key challenges that mobile developers face when it comes to securing their apps and APIs is the ability to think like an attacker. This is because attackers approach mobile app and API security from a different perspective and mindset than developers. They usually combined several techniques and chain the weaknesses and vulnerabilities of the mobile apps and their APIS to succeed on their intents.
Developers are typically focused on building functionality and features that meet user requirements, while also ensuring that the app is performant and easy to use. While security is certainly an important consideration, it is often not the primary focus of developers, who may not have a deep understanding of the various security risks that their app or API may face, and if they do they may not be aware how creative hackers can be on combining such security risks to mount a successful attack.
</.horizontal_left_card>

<.horizontal_right_card class="w-full" image_path="/images/svg/storyset_hacker-bro.svg">
In contrast, attackers are motivated by different factors, such as financial gain, political, ideological or social motives, or simply the challenge of exploiting vulnerabilities in the mobile and their APIs. They approach mobile app security and API security from a different perspective, actively seeking out weaknesses and vulnerabilities that they can exploit for their own purposes or who they work for, that can be a criminal organization, a state or just a company trying to get ahead of their competitors.
To be able to effectively secure a mobile app or API, it is therefore important for developers to be able to think like an attacker. This requires a deep understanding of the various techniques and tools that attackers use to exploit vulnerabilities in mobile apps and APIs, as well as the ability to anticipate potential attack vectors and design security controls that can mitigate these risks.
</.horizontal_right_card>

<.cta_newsletter ></.cta_newsletter>

The result:

At the moment my Phoenix 1.7 app isn’t using LiveView but I plan to do so, therefore I need to wait until I can be sure that this also works properly with LiveView tracking.

Would this be something you would consider to add support for in your Lib? If yes I can make the PR.

Exadra37

Exadra37

Wow. I didn’t expect this first class support :heart_eyes: :love_letter:

Tonight I will try your latest changes and then I will provide some feedback.

Afterwards I will try to add at least a simple quickstart example to your home page in the docs.

Where Next?

Popular in Questions Top

aadeshere1
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
New
9mm
I am constructing a JSON object (map) and I need to conditionally set a field. I’m trying to write proper elixir-way code… and I’m at a l...
New
lastday4you
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
skosch
To my knowledge, put_in, Map.update etc. all have the one limitation of not automatically creating intermediate keys when needed (for exa...
New
ovidiubadita
Hey all, I discovered Elixir and I love it. I always wanted to learn a functional programming and I intended to go for Haskell, but afte...
New
jaysoifer
Is there a way to rollback a specific migration and only that one (“skipping” all the other ones)? Would mix ecto.rollback -v 200809061...
New
vegabook
I’m brand new to Phoenix and I have stripped one of the demo applications to the bone. I just want to get an svg up on the screen. Here i...
New
pmjoe
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
New
lucidguppy
I have a super simple question about elixir - how would I take a file like this foo bar baz and output a new file that enumerates th...
New
PeterCarter
There are pre-rolled solutions for other frameworks that do work. However, Phoenix does not seem to have these. Have people had good expe...
New

Other popular topics Top

albydarned
Hello all! I am typing this post from my new MacBook Pro with the M1 chip. I’m loving it so far, and will probably use it as my daily dr...
New
lessless
I believe there are people here who are dealing with CSV files import on the daily basis, and since Excel is a really popular tool there ...
New
gshaw
What is the idiomatic way of matching for not nil in Elixir? E.g., First way: defp halt_if_not_signed_in(conn, signed_in_account) when...
New
shahryarjb
Hello, I have map which I want to convert it to string like this: the map: %{last_name: "tavakkoli", name: "shahryar"} the string I ne...
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New
AngeloChecked
What learn first? Rust or Elixir Hi Elixir community! I’m here because i want learn a new language. I’m a junior developer and mainly i ...
New
gausby
I asked this very same question on twitter and got some interesting feedback, but I thought it would be a good question to ask here as we...
1207 39297 209
New
jason.o
In the code below, if the create action is not set to accept “extra_key” as an input, it errors out with a message shown above. Is there ...
New
Qqwy
Update: How to use the Blogs &amp; Podcasts section You can post links to your blog posts or podcasts either in one of the Official Blog...
3271 126479 1222
New
vonH
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
New

We're in Beta

About us Mission Statement