ExDocMakeup - ExDoc + Makeup syntax highlighting

[tmbb]

I’ve released a package that allows you to use Makeup (my pure-elixir syntax highlighting library) to highlight the Elixir code in your ExDocs. It fixes many annoying mistakes in the default highlighting library, and adds some features, like matching delimiters; when you place the mouse cursor on top of a delimiter, such as (, the closing delimiter, in this case ), will be highlighted. Other languages will continue to use the Javascript-based syntax highlighter. You can try it right now on your hex packages. Just follow the guide:

• Detailed guide here: https://github.com/tmbb/ex_doc_makeup#exdocmakeup
• Package on hex.pm here: https://hex.pm/packages/ex_doc_makeup

Many thanks to @josevalim, who was kind enough to review my code, streamline Makeup’s dependencies and design the extension API for ExDoc that made it all possible.

It was already possible to use Makeup with ExDoc, but the configuration used to be very complex. This package simplifies all that, and is ready for use by the “general public”.

Finally, some examples of ExDocMakeup outputs:

• Official docs for ProtocolEx, by @OvermindDL1: https://hexdocs.pm/protocol_ex/readme.html (featuring extra keywords, one of the “Advanced Features”)

• Alternative docs for Ecto: https://tmbb.github.io/ecto/Ecto.html

• Alternative docs for Phoenix: https://tmbb.github.io/phoenix/overview.html

Final Notes: ExDocMakeup should be considered ready for general use. Except for the “Advanced Features”, which I explain in the guide, the API will not change, even if the underlying packages change radically. On the other hand, the API of Makeup itself might change in the future, as I find alternative uses for it. Regardless of any changes, I commit to supporting the same API for ExDocMakeup indefinitely.

[tmbb]

The focus of ExDocMakeup is always syntax highlighting, of course. But then I started wondering: I already have a custom markdown implementation included in ExDocMakeup. Currently it highlights the code. But I can use it to experiment with other features that may not be desirable for inclusion in ExDoc itself.

Besides being used for API documentation, ExDoc can also be used to author general documents about Elixir (i.e “Guides”). Guides often need to incorporate code fragments, but they can become out of sync with the code, or even contain erros that make them impossible to compile.

Enter the include directive. It allows you to include fragments of code taken from files. It’s invoked inside the markdown file. It’s a standard Earmark plugin (Earmark is the markdown implementation behind ExDoc and ExDocMakeup), and like all plugins, it must appear in a line all by itself, starting with $$. Currently I support:

Include an entire file

$$ include "lib/my_file.ex" 

Include a range of lines (inclusive)

This is inconvenient because line numbers may change if you change the contents of the file

$$ include "lib/my_file.ex", lines: 45..67 

Include a block of code:

$$ include "lib/my_file.ex", block: "my_func" 

where the block is delimited by comments in the source file:

...
# !begin: my_func
def my_func(x), do: x + x
# !end: my_func
...

I prefer the block format because unlike line numbers, it doesn’t change when you add lines above.

Configuring the language

Include some Elixir code:

$$ include "lib/my_file.ex", lines: 55..66, lang: "elixir"

Include some Python code:

$$ include "lib/external/monty.py", block: "my_python_class", lang: "python"

All options:

Currently (master branch of the GitHub repo), the include directive supports the following options, as shown above:

• :lines (Range.t) - range of lines; can’t have both :lines and :block
• :block (String.t) - block delimited by comments; can’t: have both :lines and :block
• :lang - (String.t) - programming language

The directive is a normal Elixir function call, extracted using Code.string_to_quoted and then evaluated by a mini-interpreter. This is on purpose: supporting arbitrary Elixir here doesn’t seem very smart; you’d get scoping issues and attack vectors. The current implementation is not 100% safe yet (I need to restrict further the datatypes that can be passed as arguments).

By extracting the code directly from the files, you guarantee that everything is up to date.
Besides that, you can also apply normal quality control to the code fragments (coverage, static analysis, unit testing, etc.).

Example

Suppose you have the following module doc:

defmodule ExDocMakeup do
  @moduledoc """
  ExDoc-compliant markdown processor using [Makeup](https://github.com/tmbb/makeup) for syntax highlighting.

  This package is optimized to be used with ExDoc, and not alone by itself.
  It's just [Earmark](https://github.com/pragdave/earmark)
  customized to use Makeup as a syntax highlighter plus some functions to make it
  play well with ExDoc.

  $$ include "lib/ex_doc_makeup/code_renderer.ex", block: "get_options"
  """

And the lib/ex_doc_makeup/code_rendere.ex file contains the following fragment:

...

  # !begin: get_options
  # Get the options from the app's environment
  defp get_options() do
    Application.get_env(:ex_doc_makeup, :config_options, %{})
  end
  # !end: get_options
...

When run mix docs, ExDocMakeup will fetch the appropriate fragment, and render:

Feedback?

What do you think of this API? What do you think of the block delimiters (# !begin: and # !end:)? What other features would you like to have?

In particular, @OvermindDL1, what do you think of something like this to write the upcoming Super Duper ExSpirit Tutorial? Currently we can’t use ExDocMakeup to document ExSpirit, but we can create a “dummy” Mix project where the code of the tutorial lives and then host the ExDocs on Github Pages or something like that (making it an hex package would be useless and would only take space from hex.pm)

Also pinging @josevalim. If this ends up being widely used, it can be merged into ExDoc’s core, as the underlying markdown implementation is the same as the one used by ExDoc (Earmark). In fact, this is something you can implement right now, as it’s mostly independent of Makeup and the rest of ExDoc. It yould be wise to have people experiment with ExDocMakeup first, though.

Inspiration

This feature was inspired by a similar feature in Sphinx, the main python documentation tool: Sphinx — Sphinx documentation

[Eiji]

@tmbb

What do you think of this API? What do you think of the block delimiters (# !begin: and # !end:)? What other features would you like to have?

Nope, I believe that we should not add too many comments for each helper feature - to be honest I’m generally not using any comments and I could vote to remove this feature from Elixir . Imagine that 10 projects like yours is using similar or same (!) comment type. I don’t like also example with file path and lines - especially when working in team. We should not care about something that could change. I believe that code refactoring should affect documentation as less as possible.

If it’s possible then here is better way:

defmodule MyProject do
  @moduledoc """
  <%= include MyProject.MyModule, comments: [:before_strict, :after], function: :get_options %>
  """
end
# ...
defmodule MyProject.MyModule do
  # This comment should not be added (note empty line after this comment)

  # This comment should be added
  defp get_options() do
    Application.get_env(:ex_doc_makeup, :config_options, %{})
  end
  # This comment should be added

  # This comment should also be added
end

or even simpler:

defmodule MyProject do
  @moduledoc """
  <%= include MyProject.MyModule, function: :get_options %>
  """
end
# ...
defmodule MyProject.MyModule do
  # This comment should not be added
  defp get_options() do
    # This comment should be added

    # This comment should also be added
    Application.get_env(:ex_doc_makeup, :config_options, %{})
    # This comment should be added
  end
end

For me more useful could be:

defmodule MyProject do
  @moduledoc """
  ...
  By default it's set to <%= default(:something) %>
  """
end

That should a shortcut for:

defmodule MyProject do
  @moduledoc """
  ...
  By default it's set to <%= Application.get_env(:my_project, :something) %>
  """
end

[tmbb]

@Eiji

Nope, I believe that we should not add too many comments for each helper feature

I think you’re interpreting this wrong… The goal here is not to insert code from an actually existing Elixir project (even though you can do it, of course). This is for those cases where you need some “fake” code to display inside the docs. Currently you need to write the code inline with the docs, with no way of testing it or see if it even compiles. This way, you can write “dummy” elixir modules and try to compile them or test them. The comments are meant to be used in those dummy modules, not in “real code”

@Eiji

to be honest I’m generally not using any comments and I could vote to remove this feature from Elixir

Well, that’s certainly a matter of taste… Personally, I like comments to document more complex algorithms or non-obvious interrelations between the code. I don’t like file formats that don’t allow comments (like JSON), let alone programming languages xD.

@Eiji

I don’t like also example with file path and lines - especially when working in team.

To be honest, I don’t like it either… I only added it because:

• It seems to be relatively popular in Python (subjective impression), so someone probably likes it
• It can be used to document code you don’t want to litter with these kinds of comments. This has the obvious problem that refactoring the code can mess with your docs, which is definetely undesirable

If it’s possible then here is better way:

Well, using “raw” EEx tags is not a good idea. Those files need to be processed by a markdown tool, so when do you propose I should expand the tags? Just before feeding the file to Earmark? Remember that these files will be showing literal EEx snippets quite frequently. How do you propose that I escape them? I’d need a markdown parser for it.

The way to go is to explore Earmark’s plugins, which require starting the line with $$. Anything after that line is protected from the markdown processor and fed directly to the plugin module to replace it with whatever it wants. Once you accept that the directive must be behind the $$, you don’t need to include the EEx tags: you can treat it as literal Elixir.

Being able to include the function by name is an interesting possibility, but it means I need to mess with the .beam files to know which lines I need to extract. This is probably not very hard to do, and I’ve done it before, but it creates some complexity. Again, remember the use case: We want to show examples of Elixir code, not necessarily reproduce in the docs Elixir code from the package.

Often, we need to show several function heads, separated by text paragraphs that explain something particular to that function head. I can’t single out a single head by function name… I’d need to add some markers to the source.

[tmbb]

@Eiji

For me more useful could be:

defmodule MyProject do
@moduledoc “”"
…
By default it’s set to <%= default(:something) %>
“”"
end

That should a shortcut for:

defmodule MyProject do
@moduledoc “”"
…
By default it’s set to <%= Application.get_env(:my_project, :something) %>
“”"
end

I don’t understand. Why would this be useful?

[Eiji]

@tmbb

I don’t understand. Why would this be useful?

Just because it’s shorter form and it’s for current project. I think about something like:

defmodule MyProject do
  @moduledoc """
  <% alias MyProject.DocHelper %>
  ...
  ## Feature name
  Description here ...
  Configuration info here like:
  By default it's set to: <%= inspect default(:feature_name) %>.
  <%= DocHelper.describe_default default(:feature_name) %>
  """
end

[tmbb]

You can already do that:

defmodule Dummy do
  @moduledoc """
  set to #{Application.get_env(:dummy_app, :something, "abc")}
  """
end

Load this module, open iex:

iex> h Dummy
* Dummy

set to abc

An that doesn’t work with standalone markdown pages.

[Eiji]

@tmbb

You can already do that:

defmodule Dummy do
@moduledoc “”"
set to #{Application.get_env(:dummy_app, :something, “abc”)}
“”"
end

Load this module, open iex:

iex> h Dummy

• Dummy

set to abc

An that doesn’t work with standalone markdown pages.

Yes, but I need to do it like:

defmodule MyProject.DocHelper do
  def default(name) when is_atom(name), do: Application.get_env(:my_project, name)
end
# and for each module I need to do something like:
defmodule MyProject do
  import __MODULE__.DocHelper, only: [:default]

  @moduledoc """
  #{default(:something)}
  """
end

and same for all projects

I believe that lots of projects could use this shortcut, because lots of projects are documenting its default configuration - it’s why I described such proposal.

[tmbb]

Ok, not that I can already include code blocks I’m finding it a little boring when I have to copy and paste from IEx to show IEx sessions. If I’m inside a @doc on a @moduledoc these IEx session serve as doctests, which is cool, but if I’m writing a guide in a free-form markdown file it’s just extra work. And the code can get out of sync with the docs, which is undesireable.

So I’d like to have either something like:

$$ iex> MyModule.my_func()

which would render the output inline with the docs, or a way of dumping iex sessions in a file and run that file as if they were doctest.

Is there any way of running doctests except for the “standard one”?

[Eiji]

@tmbb

I think you’re interpreting this wrong… The goal here is not to insert code from an actually existing Elixir project (even though you can do it, of course). This is for those cases where you need some “fake” code to display inside the docs. Currently you need to write the code inline with the docs, with no way of testing it or see if it even compiles. This way, you can write “dummy” elixir modules and try to compile them or test them. The comments are meant to be used in those dummy modules, not in “real code”

Ok, now I see, but still comments does not looks best for me.

@tmbb

Well, that’s certainly a matter of taste… Personally, I like comments to document more complex algorithms or non-obvious interrelations between the code. I don’t like file formats that don’t allow comments (like JSON), let alone programming languages xD.

I like this sentence: Well written code does not need any extra comments.

@tmbb

Well, using “raw” EEx tags is not a good idea. Those files need to be processed by a markdown tool, so when do you propose I should expand the tags? Just before feeding the file to Earmark? Remember that these files will be showing literal EEx snippets quite frequently. How do you propose that I escape them? I’d need a markdown parser for it.

I don’t worked with earmark plugins yet.

@tmbb

The way to go is to explore Earmark’s plugins, which require starting the line with $$. Anything after that line is protected from the markdown processor and fed directly to the plugin module to replace it with whatever it wants. Once you accept that the directive must be behind the $$, you don’t need to include the EEx tags: you can treat it as literal Elixir.

Ah, I through that $$ is your idea. I wanted to say that we should not add any extra processing if we already have it implemented in other way, but here you are saying about implementation in other library.

Here is how I’m seeing this:

# default configuration:
config :ex_doc_makeup, earmark_plugins: [], helper: MyProject.DocHelper
# ...
import ExDocMakeup, only: [:def_doc_helper]
def_doc_helper MyProject.DocHelper do
  helper :get_config_options do
    dynamic_stuff = # ...
    quote do
      # Get the options from the app's environment
      defp get_options() do
        unquote(dynamic_stuff)
        Application.get_env(:ex_doc_makeup, :config_options, %{})
      end
    end
  end
end
# ...
defmodule MyProject do
  @moduledoc """
  # ...
  $$helper get_config_options
  """
end

What do you think about it?

[tmbb]

I’m not interested in implementing that or even in merging a PR that implements this functionality, but if you like it, go ahead. Customizing Earmark is very easy.

[OvermindDL1]

@tmbb

Enter the include directive. It allows you to include fragments of code taken from files. It’s invoked inside the markdown file. It’s a standard Earmark plugin (Earmark is the markdown implementation behind ExDoc and ExDocMakeup), and like all plugins, it must appear in a line all by itself, starting with $$. Currently I support:

Whooooo awesome!

This is definitely something that I can get use out of. ^.^

@tmbb

In particular, @OvermindDL1, what do you think of something like this to write the upcoming Super Duper ExSpirit Tutorial? Currently we can’t use ExDocMakeup to document ExSpirit, but we can create a “dummy” Mix project where the code of the tutorial lives and then host the ExDocs on Github Pages or something like that (making it an hex package would be useless and would only take space from hex.pm)

I’d already been thinking of making a subdirectory just to make docs for it and have it output to ../docs instead of ./docs. ^.^

Can just alias the docs mix command to shell out to that subdirectory’s mix docs.

@tmbb

Being able to include the function by name is an interesting possibility, but it means I need to mess with the .beam files to know which lines I need to extract. This is probably not very hard to do, and I’ve done it before, but it creates some complexity. Again, remember the use case: We want to show examples of Elixir code, not necessarily reproduce in the docs Elixir code from the package.

Could just string_to_quoted the model and parse the ast for the location information?

@Eiji

Just because it’s shorter form and it’s for current project. I think about something like:

Hmm I could see the use in calling back into the ex_doc system to acquire information from elsewhere ‘too’.

@tmbb

but if I’m writing a guide in a free-form markdown file it’s just extra work. And the code can get out of sync with the docs, which is undesireable.

Yeah this is an issue I have too, I like to bring in my tests into docs but cannot always have a place to put in doctests (nor can I in many cases very easily…).

@Eiji

Ah, I through that $$ is your idea. I wanted to say that we should not add any extra processing if we already have it implemented in other way, but here you are saying about implementation in other library.

Yeah that is a requirement of earmark that ex_doc itself uses.

[tmbb]

@OvermindDL1

Could just string_to_quoted the model and parse the ast for the location information?

Too brittle. Would only wprk with functions defined with def, defmacro, etc Also, often you want to show a specific head. This seems like too much work for very little benefit just to avoid adding two comments to the source file.

EDIT: other issues: if the function has a typespec, would it include the typespec? Would there be a mechanism to include callbacks too? There are lot’s of choices I’d have to make.

[tmbb]

But if anyone sends a PR I’ll probably merge it.

[tmbb]

New version released (v0.3.0). The API is the same but it now uses the new (much faster!) elixir lexer.
Feel free to try it on your packages.

[tmbb]

Makeup will be included as the default syntax highlighter in ExDoc 0.19. This package is no longer relevant.