Test as First Class Documentation (ExDoc)

alchemist_ubi · July 31, 2018, 3:24pm

I taught my friend to use the testing as the tool for understanding and knowing how to use the packages that I built.

Instead of asking me how it works I just send him to the Unit Tests so he can see how I tested it and by so he could understand how to use it.

I like the idea of treat tests as documentation directly into ExDoc generated documentation.

Some of these packages you will find things like https://github.com/straw-hat-team/straw_hat_review/blob/master/test/straw_hat_review/aspects_test.exs

defmodule StrawHat.Review.AspectsTest do
 describe "find_aspect/1" do
    test "with valid id should find the aspect" do
      aspect = insert(:aspect)

      assert {:ok, _aspect} = Aspects.find_aspect(aspect.id)
    end

    test "with invalid id shouldn't find the aspect" do
      assert {:error, _reason} = Ecto.UUID.generate() |> Aspects.find_aspect()
    end
  end
end

Notice that I am using the function name with the arity as some sort of pointer to the function so the users can know the scope of my test is around that function.

I could remove however that from the message when the test is not related to a specific function and at that moment probably becomes part of the module scope from the documentation perspective.

If add some metadata to the testing and combine it with a description that represents some kind of use case I would like to see it on my ExDoc as well.

For example,

defmodule StrawHat.Review.AspectsTest do
  # whatever other way, just for showing the idea
 @doc function: StrawHat.Review.find_aspect/1
 describe "find_aspect/1" do
    test "with valid id should find the aspect" do
      aspect = insert(:aspect)

      assert {:ok, _aspect} = Aspects.find_aspect(aspect.id)
    end

    test "with invalid id shouldn't find the aspect" do
      assert {:error, _reason} = Ecto.UUID.generate() |> Aspects.find_aspect()
    end
  end
end 
# whatever other way, just for showing the idea
@doc module: StrawHat.Review
test "..." do end

This way the documentation could pull out the test with the description and use that for generate a new section on the documentation.

Maybe some collapsed blocks where you can see the description and expand the block so see the whole unit test.

Anyway, I hope you get the idea of what I am suggesting, I would love to take advantage of the testing for generating documentation.

cc: @josevalim

OvermindDL1 · July 31, 2018, 6:00pm

Isn’t that already doctests though?

alchemist_ubi · July 31, 2018, 6:23pm

No really, doctest allow you to test from documentation but the issue is that you can’t use any macros or any helper function and so on because you dont have any of that in the doctest.

Unless I am talking non-sense, in my case, I have ex_machine on it.

OvermindDL1 · July 31, 2018, 6:29pm

Doctests should run in the context of the ExUnit module that they were brought in to, and as such you can call functions exposed or defined in that module, just like Phoenix eex templates can call anything exposed in its corresponding View module.

alchemist_ubi · July 31, 2018, 7:45pm

Naaahhh

hhhmmmmmm I am an idiot!

But still, I wouldn’t put all the unit testing inside the Doctests.

LostKobrakai · July 31, 2018, 7:54pm

Doctests are getting less and less useful the more and more setup they need imho. See e.g. the documentation for Ecto.Changeset. They’re plain text docs with no doctests. So doctests are certainly not the holy grail to bringing tests/docs together. I feel like they should stay a tool of ensuring docs are correct where possible.

Ankhers · July 31, 2018, 7:56pm

I do not think pointing someone to your tests is the proper way to get them to understand how to use your package. I think better written documentation is the answer, especially if I am relatively new to the language / platform.

alchemist_ubi · July 31, 2018, 8:13pm

That was my understanding about it. Especially that becomes harder to use Doctests when you have multiple lines of code for the test.

Documentation is not for describing every possible situation that happens on the package.

You could see all the possible situations that the people wrote and understand how the package work by looking the testing file.

And just worth to mention, I am not talking about no writing documentation but enhance it.

Also, the older I become the more I try to train people to do that. Engineers do not like to read texts but we like to read the code.

I found interesting that my friend and another friend from school who didn’t code on Elixir at all do that habit before anything. And for me becomes easier since they could take a look to the testings for figure out how to do specific task related to the package.

OvermindDL1 · July 31, 2018, 8:17pm

Exactly my point, put the ‘understanding’ usage of the API in the doctests, put the grunt and setup work in the ExUnit script and just call to it.

peerreynders · July 31, 2018, 8:19pm

That extracts code from the documentation.

This approach imports example code from the tests into the documentation.

The standard Go testing facility supports this testing/documentation approach with Examples.

alchemist_ubi · July 31, 2018, 8:24pm

LOL I missed understood what you meant

I like that idea!

But the issue with this is that I am trying to push my testing as documentation, like flip the coin, even when that approach would work I agree with @LostKobrakai

But at the same time I want my testing to be part of the guideline of the package, I find that people love to read the testing for understanding things rather than asking to people.