Implementing custom Markdown parser with the MD Library

Guilty.

The docs of md must be updated and extended widely.

In the first place, you’ve found a bug in the custom parsers implementation, thanks for that. Fixed in v0.9.7.

Second, you kinda mix up custom parsers and syntax. The very correct change in config.exs would now be handled properly, the correct syntax is:

import Config

config :md, syntax: %{
  custom: [{"<.", {MasWeb.MdParser, %{}}}]
}

The custom parser is an implementation of Md.Parser behaviour, so copy-pasting from the library source wouldn’t help much. Instead, you are supposed to implement the whole parser. The very naïve implementation would look like this:

defmodule MasWeb.MdParser do
  alias Md.Parser.State

  @behaviour Md.Parser

  @impl true
  def parse(input, state) do
    # <.card image_path="/images/awesome.svg">
    #   Some nice card with an image on the left.
    # </.card>
    # Continuing after the card.

    [tag, rest] = String.split(input, " ", parts: 2)
    [content, rest] = String.split(rest, "</.#{tag}>", parts: 2)
    {rest, %State{state | ast: [content | state.ast]}}
  end
end

Once we use <. as a tag, <. would not be passed to the handler itself. Hence we got card in the first split and the inner tag content in the second split. We end up with rest, which must be passed back to the “main” parser as a continuation, and content which you should translate to AST yourself (because it’s a custom parser.) Here we simply return back a text node.

Sidenote: custom name is essential, common would not work, it’s used as a tag type.

Thanks for giving the library a try, please, don’t hesitate to ask if anything. I’d love to make some progress with its docs and tests, but unfortunately, it serves our current needs and I am like, ok, later, then

Wow. I didn’t expect this first class support

Tonight I will try your latest changes and then I will provide some feedback.

Afterwards I will try to add at least a simple quickstart example to your home page in the docs.

I was under impression that is the Elixir community standard.

Package published to md | Hex
(6e45712029ecf10f552c16a7836259f92a132f9ce4235c0937598714cd53f87b)

I gave a thought to your usage example and now I have a question.

md already supports tags via tag: syntax, but the support is very limited (no attributes whatsoever,) I am having plans to extend it, and that’s why I’d love to see a real example of how it could be used.

My proposal would be for attributes to go to resulting attributes as is, the default parser would do all the work, produce an AST, and the tag itself then be passed to the (newly introduced, optional) transformer.

Before I am to file an issue and start working on it, I’d love to know what do you expect to get back as the result.

To answer this directly: I don’t know yet.

What I am trying to achieve it to embed any HEEX template in a markdown document. This would allow me to reuse the same components used elsewhere in other HEEX templates,

For example:

• cards - will contain text with or without images, buttons, etc. Text would be wirrtten in markdown.
• forms - e.g. to subscribe to a newsletter, for a poll.
• tabs - be able to provide content as tabs
• tables - Allow to add nicely formatted tables

All of this HEEX components would be customisable in order to pass classes and any other html attributes, etc.

The trigger was to be able to write my website pages in markdonw and have them with text and images alternating:

image1139×807 90 KB

This current webpage is written in markdonw and content aligned with a lot of css trickery and repetition of images declaration:

# Hack Yourself First {:.markdown-header}

![hacker_mindset](/images/svg/storyset_mobile-encryption-amico.svg){:.lg:markdown-p-img .md:markdown-p-img .markdown-p-img} One of the key challenges that mobile developers face when it comes to securing their apps and APIs is the ability to think like an attacker. This is because attackers approach mobile app and API security from a different perspective and mindset than developers. They usually combined several techniques and chain the weaknesses and vulnerabilities of the mobile apps and their APIS to succeed on their intents.
Developers are typically focused on building functionality and features that meet user requirements, while also ensuring that the app is performant and easy to use. While security is certainly an important consideration, it is often not the primary focus of developers, who may not have a deep understanding of the various security risks that their app or API may face, and if they do they may not be aware how creative hackers can be on combining such security risks to mount a successful attack.

![hacker_mindset](/images/svg/storyset_hacker-bro.svg){:.markdown-p-img .sm:hidden}In contrast, attackers are motivated by different factors, such as financial gain, political, ideological or social motives, or simply the challenge of exploiting vulnerabilities in the mobile and their APIs. They approach mobile app security and API security from a different perspective, actively seeking out weaknesses and vulnerabilities that they can exploit for their own purposes or who they work for, that can be a criminal organization, a state or just a company trying to get ahead of their competitors. 
To be able to effectively secure a mobile app or API, it is therefore important for developers to be able to think like an attacker. This requires a deep understanding of the various techniques and tools that attackers use to exploit vulnerabilities in mobile apps and APIs, as well as the ability to anticipate potential attack vectors and design security controls that can mitigate these risks.![hacker_mindset](/images/svg/storyset_hacker-bro.svg){:.lg:markdown-p-img .md:markdown-p-img .markdown-p-img .sm:show .hidden}

Writing markdonw like this becomes tedious and time consuming. Earmark is being used to parse it.

Then I also came to realise that I could render HEEX templates with Earmark, but not pass custom content to the HEEX templates, and this was when I started to look into your library.

By developing the ability to think like an attacker, mobile developers can become more proactive in identifying and addressing security risks in their apps and APIs. This can help to enhance the overall security of the app or API, and ensure that user data is protected from potential threats. 

<%= cta_newsletter(assigns) %>

That renders to this:

image1121×587 31.6 KB

The HEEX template that I am currently trying to use from the markdown to include the card with text and image:

<div>
  <a href="#" class="flex flex-col items-center bg-white border border-gray-200 rounded-lg shadow md:flex-row md:max-w-xl hover:bg-gray-100 dark:border-gray-700 dark:bg-gray-800 dark:hover:bg-gray-700">
      <img class="object-cover w-full rounded-t-lg h-96 md:h-auto md:w-48 md:rounded-none md:rounded-l-lg" src={assigns[:image_path]} alt="">
      <div class="flex flex-col justify-between p-4 leading-normal">
        <p class="mb-3 font-normal text-gray-700 dark:text-white">
          <%= render_slot(assigns[:inner_block]) %> 
        </p>
      </div>
  </a>
</div>

The Card HEEX template it’s only a draft. I still need add variables to customise the HTML attributes.

Let me know if you have further questions.

Thank you, that helped a lot.

Now I understand it’s surely not about tags per se.

The main problem to figure out would be who leads the parsing. EEx allows custom engine implementation and one might call a markdown parser from a custom EEx.Engine.handle_text/3 callback.

We can support eex/heex as is, or introduce delegate which would be like custom but instead of the implementation of Md.Parser it would rather parse it until it’s closed, maybe apply its own format, and then delegate the whole to the external engine.

The question who leads the process remains though. Consider <%= cta_newsletter(assigns) %> where cta_newsletter/1 returns "Subscribe to **the newsletter**" string, or <.card image=...>My _fancy_ text with **markup**</.card>.

Anyway, it needs some time to twiddle the puzzle in hands, but meanwhile, I would suggest you try the vice-versa approach with a HEEx engine ruling the process and md being called on text nodes (no idea of how to achieve that, though, HEEx does not seem to be friendly to external parsers.

Thank you very much for the options you gave me to think about. I will tak a look to them for sure.

Yesterday night I made it work and it seems that it matches the approach you suggest here:

My code:

defmodule MasWeb.MdParser.Heex do

  alias Md.Parser.State

  @behaviour Md.Parser

  @impl true
  def parse(input, state \\ %State{})
  def parse(input, state) do
    # @TODO handle the case for a tag without attributes:
    #       * <.whatever/>
    #       * <.whatever>content</.whatever>
    [tag, rest] = String.split(input, " ", parts: 2)
    [content, rest] = String.split(rest, "</.#{tag}>", parts: 2)
    [attrs, content] = String.split(content, ">", parts: 2)

    # parses markdown inside an HEEX tag to HTML and then rebuild the HEEX tag
    # to allow for the Phoenix.LiveView.HTMLEngine to the parse it as a regular 
    # *.heex file, thus keeping all the niceties for LiveView tracking? (need to
    # double check my assumption)
    html = Md.Parser.generate(content)
    html = "<.#{tag} #{attrs}>#{html}</.#{tag}>"

    {rest, %State{state | ast: [html | state.ast]}}
  end
end

The custom Phoenix Engine:

defmodule MasWeb.PhoenixEngine do

   # @link Inspired by: https://github.com/boydm/phoenix_markdown/blob/ef7b5f76f339babec688021080a70708d9ddf1c1/lib/phoenix_markdown/engine.ex#L22

  @moduledoc """
  a single public function (compile) that Phoenix uses to compile incoming templates. You should not need to call it yourself.
  """

  @behaviour Phoenix.Template.Engine

  @doc """
  Callback implementation for `Phoenix.Template.Engine.compile/2`

  Precompiles the String file_path into a function defintion, using the EEx and Earmark engines

  The compile function is typically called for by Phoenix's html engine and isn't something
  you need to call your self.

  ### Parameters
    * `path` path to the template being compiled
    * `name` name of the template being compiled

  """
  def compile(path, _name) do

    options = [
      engine: Phoenix.LiveView.TagEngine,
      file: path,
      line: 1,
      caller: __ENV__,
      source: "",
      tag_handler: Phoenix.LiveView.HTMLEngine
    ]

    path
    |> File.read!()
    |> Md.generate(Md.Parser.Default, format: :none)
    |> EEx.compile_string(options)
  end
end

The config.exs:

config :phoenix, :template_engines,
  # will handle all markdown files that have an extension *.html.md, e,g. test.html.md
  md: MasWeb.PhoenixEngine

config :md, syntax: %{
  custom: [{"<.", {MasWeb.MdParser.Heex, %{}}}]
}

Add md extension to Phoenix live reload in config/dev.exs

config :mas, MasWeb.Endpoint,
live_reload: [
  patterns: [
    ~r"priv/static/.*(js|css|png|jpeg|jpg|gif|svg)$",
    ~r"priv/gettext/.*(po)$",
    ~r"lib/mas_web/(controllers|live|components)/.*(ex|heex|md)$"
  ]
]

The markdown file test.html.md:

## TEST

<.horizontal_left_card class="bg-transparent" image_path="/images/svg/storyset_mobile-encryption-amico.svg">
One of the key challenges that mobile developers face when it comes to securing their apps and APIs is the ability to think like an attacker. This is because attackers approach mobile app and API security from a different perspective and mindset than developers. They usually combined several techniques and chain the weaknesses and vulnerabilities of the mobile apps and their APIS to succeed on their intents.
Developers are typically focused on building functionality and features that meet user requirements, while also ensuring that the app is performant and easy to use. While security is certainly an important consideration, it is often not the primary focus of developers, who may not have a deep understanding of the various security risks that their app or API may face, and if they do they may not be aware how creative hackers can be on combining such security risks to mount a successful attack.
</.horizontal_left_card>

<.horizontal_right_card class="w-full" image_path="/images/svg/storyset_hacker-bro.svg">
In contrast, attackers are motivated by different factors, such as financial gain, political, ideological or social motives, or simply the challenge of exploiting vulnerabilities in the mobile and their APIs. They approach mobile app security and API security from a different perspective, actively seeking out weaknesses and vulnerabilities that they can exploit for their own purposes or who they work for, that can be a criminal organization, a state or just a company trying to get ahead of their competitors.
To be able to effectively secure a mobile app or API, it is therefore important for developers to be able to think like an attacker. This requires a deep understanding of the various techniques and tools that attackers use to exploit vulnerabilities in mobile apps and APIs, as well as the ability to anticipate potential attack vectors and design security controls that can mitigate these risks.
</.horizontal_right_card>

<.cta_newsletter ></.cta_newsletter>

The result:

image768×820 65.5 KB

At the moment my Phoenix 1.7 app isn’t using LiveView but I plan to do so, therefore I need to wait until I can be sure that this also works properly with LiveView tracking.

Would this be something you would consider to add support for in your Lib? If yes I can make the PR.

Sidenote: Handling a tag with and without attributes by the same code should be as easy as swapping splits

What worries me, is generating of the HTML where it should not technically be generated.

I am not sure it would work properly with nested tags, although it seems it would, still it looks like a kludge. Md.Parser supports attaching a listener, which might modify the result, and we should at least re-pass it to the underlying generate/2. Also, it’d fail on creating deferred links ([foo][1] followed by [1] link,) and lose the context in general, when not surrounded by \n\n.

md is a streaming parser, unlike [H]EEx, which makes it less trivial to interoperate with other engines.

Anyway, I will think about how can it be done without leaving AST representation and come back to you.

I’m really interested in rendering function components inside markdown files.

Any advances in this front?

Honestly, there is not much demand and I did not come up with a generic solution.

If you could share an example of what you need, I could give it another spin.