EDT: Elixir Documentation Tool

[Eiji]

Hey, I’m now planning EDT (Elixir Documentation Tool) - a documentation tool for all Elixir projects.

Planned features:

1. Language support - I will define API and 2-3 implementations for it by default:
   a) Elixir support
   b) JavaScript support (probably based on jsdoc)
   c) maybe also CoffeeScript support
   Maybe also support for NIFs and erlang code?
   Imagine that your project that uses (for example): rustler, some erlang code, lots of JavaScript code and of course lots of Elixir code will now be documented in one clear page with same look!

2. Grouping support - Each language implementation should provide also implementation for grouping code. For example ex_doc now groups Elixir modules into:
   a) Exceptions
   b) Protocols
   c) Tasks
   d) Rest - normal modules
   I’m planning to implement much more + add ability to define your own module types in case you are writing your own macros that defines custom modules.
   Maybe Elixir core team will decide to support one unique attribute that selects specified module type like: @module_type :protocol or: @module_type {:core, :protocol} (auto generated when creating protocol).

3. Sub-grouping support - Each group (in Elixir module type) should provide implementation for grouping it’s contents. Currently in ex_doc are separated:
   a) Callbacks
   b) Functions
   c) Types
   Similarly to 2nd point - I will also add ability to define your own. For example ExApi will sort implemented features (in implementation module) by it’s implementation status (i.e. :supported, :not_supported and :not_implemented).

4. ExDoc features like ability to choose markdown tool and themes or define extra pages.

5. Support for i18n - by default all documentation must be written in base language (probably all projects will write documentation in English language). I will write an small Phoenix-based local server with (probably) SQLite database to add translation for documentation. Tool will not generate documentation parsing source files - it will parse source files and saves all data into database in order to translate it in future. Then on that site you will be able to generate and preview your documentation.

6. Developer mode - I will add ability to document private functions and I will add simple switcher that shows/hides documentation for them. By default in developer mode documentation will show CONTRIBUTING.md file and README.md file in normal mode.

7. Save your settings - imagine that you only need to switch theme one time and then only click save your settings and choose for all projects!

8. Other enhancements:
   a) use latest Ecmascript and HTML5 api - for example I could generate one big file with simple HTML templates
   b) new sidebar - I have my own version of sidebar somewhere on my hard drive. Let me surprise you.
   c) more and more settings
   and much, much more

9. Your ideas - of course I’m always open to your ideas!

I will probably start work on this after 1.0.0 release of ExApi. These projects will complement each other. EDT will defines apis using ExApi library as this will be great example of this library usage. Also ExApi will use EDT to generate special documentation.

What do you think about it?

[ShalokShalom]

LiveScript is the successor for Coffeescript?

[OvermindDL1]

That looks quite cool! I’m interested to see how it will develop!

Definitely use @tmbb’s code highlighter by default, it is far more accurate!

[tmbb]

Some questions:

How will this integrate with the hexdocs.pm? As far as I know, the documentation pipeline is quite inflexible right? Don’t you have to use ExDoc, and in particular a published version of ExDoc? Even assuming you manage to host documentation produced by EDT on hexdocs, will the function head anchors be compatible with ExDoc, i.e. Can I reference a project documented by EDT from a project documented by ExDoc? If not, then please think very hard about this… One thing that makes hexdocs so good is that you can trivially write links between different projects. Break that and you’re into Python-land: they have a great documentation tool, sphinx (which is light-years ahead of something like ExDocs), but no centralized website for documentation, and it’s very inconvenient to link from a project to another. If you break link compatibility between projects I’ll never use EDT and tell everyone note use it.

Why would you use something like SQLite? Is there anything that SQLite gives you that you can’t have with native BEAM tools, such as Mnesia? Maybe you want to do something with full-text search, which I think the BEAM options don’t support… Just evaluate your options and don’t choose SQLite just because that’s what everyone else is using.

Now, my opinion on certain topics:

@Eiji

JavaScript support (probably based on jsdoc)

You can probably take a look at how Sphinx does it (there is an extension that supports javascript). If everything else fails, you can always fallback to node-based options. Same for documenting rust code - maybe you could leverage some cargo tooling?

In general, Sphinx is a good source of inspiration for something like this: it does everything you want (i18n, arbitrary pages, freeform documentation, arbitrary groupings, etc.) and more. The problem is that it’s written in (strange?) Python and impossible to use as a library.

Finally, talk to @josevalim about this. There’s the possibility that he might be interested in integrating some of these things into ExDoc, and the ExDoc team is open to PR with new features, as long as you have a real use case.

Regarding my syntax highlighter (makeup). You’d have to write some lexers, but it can highlight pretty much everything. It avoids certain mistakes almost all Elixir highlighters make. I’m currently restructuring some internal details, but the API will probably become stable except for one or two details. But I don’t know if it’s ready for public consumption besides using it with my markdown library makedown (basically Dave’s earmark with makeup to highlight the code).

[Eiji]

@tmbb

How will this integrate with the hexdocs.pm?

I don’t know how everything around hexdocs.pm works, but if there is only need to write documentation to doc directory then this should not be a problem.

@tmbb

Even assuming you manage to host documentation produced by EDT on hexdocs, will the function head anchors be compatible with ExDoc, i.e. Can I reference a project documented by EDT from a project documented by ExDoc? If not, then please think very hard about this… One thing that makes hexdocs so good is that you can trivially write links between different projects. Break that and you’re into Python-land: they have a great documentation tool, sphinx (which is light-years ahead of something like ExDocs), but no centralized website for documentation, and it’s very inconvenient to link from a project to another. If you break link compatibility between projects I’ll never use EDT and tell everyone note use it.

Hmm, I was not think about it yet. I could add a special file to detect if linked project uses EDT (file exists) or ExDoc (file not exists). Then by little changing contents of functions that adds links into text (from ex_doc source) I could link to EDT or ExDoc documentation.
This is definitely good idea to add on todo list.

@tmbb

Why would you use something like SQLite? Is there anything that SQLite gives you that you can’t have with native BEAM tools, such as Mnesia? Maybe you want to do something with full-text search, which I think the BEAM options don’t support… Just evaluate your options and don’t choose SQLite just because that’s what everyone else is using.

I need a relational database and I don’t need so much features like in PostgreSQL - this will be used only by one user. I will save SQLite database to project folder (just simple database without authentication).

@tmbb

You can probably take a look at how Sphinx does it (there is an extension that supports javascript). If everything else fails, you can always fallback to node-based options. Same for documenting rust code - maybe you could leverage some cargo tooling?

In general, Sphinx is a good source of inspiration for something like this: it does everything you want (i18n, arbitrary pages, freeform documentation, arbitrary groupings, etc.) and more. The problem is that it’s written in (strange?) Python and impossible to use as a library.

Thanks for recommendation - I will take a look. Maybe I wil find some really interesting ideas there that I don’t have on my todo list. Python is strange for me too. Somehow I don’t like it from first saw.

@tmbb

Finally, talk to @josevalim about this. There’s the possibility that he might be interested in integrating some of these things into ExDoc, and the ExDoc team is open to PR with new features, as long as you have a real use case.

If I good remember core team is not interested in at least i18n. One time I asked about translating official Elixir page. I also saw that documentation for private functions is also not a priority for ExDoc.

@tmbb

Regarding my syntax highlighter (makeup). You’d have to write some lexers, but it can highlight pretty much everything. It avoids certain mistakes almost all Elixir highlighters make. I’m currently restructuring some internal details, but the API will probably become stable except for one or two details. But I don’t know if it’s ready for public consumption besides using it with my markdown library makedown (basically Dave’s earmark with makeup to highlight the code).

It’s best syntax highlighter I saw, so of course I’m interested. I would also like to write my own styles for it and maybe allow add custom styles for user into any local storage (if I good remember there are few options to save data locally).

[Eiji]

@ShalokShalom

LiveScript is the successor for Coffeescript?

In Polish version of Wikipedia I can read that it’s old name for JavaScript, hahah

I’m still unsure in implementing other front-end language support than JavaScript, but no need to worry! You will be able to write a library that implements every language that you need or just make PR!

[Eiji]

@OvermindDL1

Definitely use @tmbb’s code highlighter by default, it is far more accurate!

Fully agree! I will also write my own styles for it!

[tmbb]

@Eiji

could link to EDT or ExDoc documentation.

I think you’re getting this wrong. You should make sure your function heads have HTML anchors and that the URL for those anchors is the same as it would be if they were generated by ExDoc. Otherwise it’s impossible for my project that uses ExDoc to link to your project that uses EDT.

[michalmuskala]

The only thing http://hexdocs.pm needs is a doc folder with index.html. There’s no requirement whatsoever to use ex_doc. There are packages using erlang’s edoc, for example: https://hexdocs.pm/sbroker/

[Eiji]

@tmbb

I think you’re getting this wrong. You should make sure your function heads have HTML anchors and that the URL for those anchors is the same as it would be if they were generated by ExDoc. Otherwise it’s impossible for my project that uses ExDoc to link to your project that uses EDT.

If I will have more features and more links then sooner or later you will have to add your own links. How should I support for example adding link to supported features (in documentation for x api implementation when you are using ExDoc)?
I will keep a compatibility as long as it will not limit my implementation.
Optionally I could add a PR to support links to EDT in ExDoc by extra configuration if there will be any need for doing that. I believe that if it will be optionally then it should not be a problem for ExDoc maintainers.

[Eiji]

@michalmuskala: Thanks for information!

[ShalokShalom]

@Eiji

In Polish version of Wikipedia I can read that it’s old name for JavaScript, hahah

I mean http://livescript.net

So, why Coffeescript when LiveScript is there?

[Eiji]

@ShalokShalom

I mean http://livescript.net

So, why Coffeescript when LiveScript is there?

I know and already saw this site.
As I already wrote:

@Eiji

I’m still unsure in implementing other front-end language support than JavaScript, but no need to worry! You will be able to write a library that implements every language that you need or just make PR!

For example:

import ExApi.Implementation

alias EDT.Apis.{Group, Language, SubGroup}

def_api_impl EDT.Apis.Language, :live_script do
  def_feature_impl collect_data do
    # get all LiveScript documentation
    # group it and for each group add sub-groups
  end

  # @impl.id -> :live_script
  def_feature_impl name, do: Macro.camelize(@impl.id)

  # ...
end

def_api_impl Group, :class do
  @impl_group :live_script

  def_feature_impl get_data_label(group), do: Macro.camelize(group.id)

  # implement label for every language you want to support in your documentation
  def_feature_impl get_label(:en), do: "Classes"

  # ...
end

def_api_impl SubGroup, :static_class_property do
  @impl_group :live_script

  def_feature_impl get_data_label(sub_group), do: Macro.camelize(sub_group.id)

  # implement label for every language you want to support in your documentation
  def_feature_impl get_label(:en), do: "Static properties"

  # ...
end