Turn module docs and specs into a json definition?

Questions / Help
, ,
1

Given the following code:

defmodule MathOps do
  @moduledoc """
  This module provides math operations
  """

  @doc """
  This function returns the sum of input

  """
  @spec add(integer, integer) :: map
  def add(a, b) do
    %{"result" => Integer.to_string(a + b)}
  end
end

I’d like to load the file and turn it into a json spec like so:

{
    "module": "MathOps",
    "doc": "This module provides math operations",
    "defs": [
        {
            "name": "add",
            "doc": "This function returns the sum of input",
            "params": [
                {"name": "a", "type": "integer", "default": null},
                {"name": "b", "type": "integer", "default": null}
            ],
            "return_type": "map"
        }
    ]
}

Any ideas/guidance would be greatly appreciated :slight_smile:

2

Most of this information is also used in ExDoc templates (used to generate docs like on hexdocs.pm) so that tool’s source could be a good place to start:

2 Likes
3

Read up on ExDoc a bit and here is my hacky attempt to get a quick win:

module = Module.concat(["MathOps"])
config = ExDoc.Config.build("Elixir", "1", [source_beam: "beam_dir"])
ExDoc.Retriever.docs_from_modules([module], config)

It returns an empty list however :confused:

Any ideas how to get a module info dump (hopefully with attribute details) from ExDoc?

4

Did some more reading. ExDoc is using Code.fetch_docs under the hood it seems inside defp docs_chunk. Cool I give it a try:

Code.ensure_loaded?(MathOps)
# true
Code.fetch_docs(MathOps)
# {:error, :module_not_found}

Bummer. Well after more reading I find this: Module documentation not immediatelly available after ensuring then module is compiled - #3 by josevalim. So I go down another rabbit hole trying to ensure all modules are done compiling etc get raw beam file etc and it feels overly complicated now. Hopefully someone can provide insight.

5

I wrote docout to accomplish something very similar. Not sure I understand the issues you’ve run into so far, but maybe it (or the source) could help.

6

That looks great! If I can’t find a vanilla Elixir way I’ll give it a go.

In regards to Code.fetch_docs I also found this snippet on the writing documentation page:

Code.fetch_docs/1

Elixir stores documentation inside pre-defined chunks in the bytecode. It can be accessed from Elixir by using the Code.fetch_docs/1 function. This also means documentation is only accessed when required and not when modules are loaded by the Virtual Machine. The only downside is that modules defined in-memory, like the ones defined in IEx, cannot have their documentation accessed as they do not have their bytecode written to disk.

So I have to find a way to write the module to beam vm disk, then pass the beam file to Code.fetch_docs/1.

Still trying to reverse engineer Exdoc to see how it does this automagically :stuck_out_tongue:

7

Any module from dependencies or your lib folder will have the Docs chunk. You can try any Elixir module to get started Code.fetch_docs(String).

1 Like
8

From where / how do I run that though? Previous attempts give:

Code.fetch_docs(MathOps)
# {:error, :module_not_found}

I tried fetch_docs from livebook/local iex/ load + compile module first then fetch etc. I guess I need ELI5 :laughing:

9

When are you trying to call fetch_docs ? You may need to ensure that compilation of your app source has finished. In docout that is accomplished here: docout/docout.ex at 5a0ebd0a1cbb77bc9d82b6efd2d0d97d24ebb960 · tfwright/docout · GitHub

10

Here is local iex:

iex(3)> import Code
Code
iex(4)> Code.compile_file("./MathOps.ex")
[
  {MathOps,
   <<70, 79, 82, 49, 0, 0, 7, 112, 66, 69, 65, 77, 65, 116, 85, 56, 0, 0, 0,
     197, 0, 0, 0, 21, 14, 69, 108, 105, 120, 105, 114, 46, 77, 97, 116, 104,
     79, 112, 115, 8, 95, 95, 105, 110, 102, 111, 95, ...>>}
]
iex(5)> MathOps
MathOps
iex(6)> MathOps.add(1,2)
%{"result" => "3"}
iex(7)> 
nil
iex(8)> Code.ensure_loaded?(MathOps)
true
iex(9)> Code.fetch_docs(MathOps)
{:error, :module_not_found}
iex(10)>

:smiling_face_with_tear:

One thing I have not tried yet is Mix.Task.Compile

11

Geewiz I finally figured it out and as expected it was something simple. I was starting my local iex session by calling iex directly without any params.

I started iex as iex -S mix and it worked perfectly.

And now fetch_docs actually works hooray!

iex(3)> MathOps.add(1,2)
%{"result" => "3"}
iex(4)> Code.fetch_docs(MathOps)
{:docs_v1, 2, :elixir, "text/markdown",
 %{"en" => "This module provides math operations\n"}, %{},
 [
   {{:function, :add, 2}, 6, ["add(a, b)"],
    %{"en" => "This add function returns the input sum\n\n"}, %{}},
   {{:function, :minus, 2}, 15, ["minus(a, b)"], :none, %{}}
 ]}

And for bonus points I got Exdoc working as well (which provides module specs):

config = ExDoc.Config.build("Elixir", "1", [source_beam: "beam_dir"])
docs = ExDoc.Retriever.docs_from_modules([MathOps], config)

Now I have the info I need to build the json specification :slight_smile:

12

Here is v1 prototype to get a json spec from module docs. I also learned about running iex> c("MathOps.ex", ".") to compile the file.

defmodule MathOps do
  @moduledoc """
  This module provides math operations
  """
  @after_compile {__MODULE__, :_build_json_def}
  def _build_json_def(_env, bytecode) do
    {:docs_v1, _, _, _, module_doc, _, docs} = Code.fetch_docs(__MODULE__)
    {:ok, specs} = Code.Typespec.fetch_specs(bytecode)
    defs =
      for {{_, doc_name, _}, _, [sig], doc, _} <- docs,
        {{spec_name, _}, [{:type, _, :fun, [{:type, _, :product, args}, {_, _, result, _}]}]} <- specs,
        doc != :hidden,
        doc_name == spec_name do
          params =
            for arg <- args,
            {:type, _, type, _} = arg do
              %{"type" => Atom.to_string(type)}
            end
          %{"name"=> Atom.to_string(doc_name), "doc"=> doc, "signature"=> sig, "params"=> params, "return"=> result}
      end
    document = %{"module"=> Atom.to_string(__MODULE__), "doc"=> module_doc["en"], "defs"=> defs}
    IO.inspect(document)
  end

  @doc """
  This function adds
  """
  @spec add(integer, integer) :: map()
  def add(a, b) do
    %{"result" => Integer.to_string(a + b)}
  end

end

Looks kinda bad in my opinion but it works :stuck_out_tongue: . On compile it spits out clean looking map. Couple minor things yet and then json dump to file.

%{
  "module" => "Elixir.MathOps",
  "doc" => "This module provides math operations\n",
  "defs" => [
    %{
      "doc" => %{"en" => "This function adds\n"},
      "name" => "add",
      "params" => [%{"type" => "integer"}, %{"type" => "integer"}],
      "return" => "map",
      "signature" => "add(a, b)"
    }
  ]
}

Thinking later on this could be a mix compile task but haven’t learned about it yet.