Doctest documentation vs testing in a mix project

Questions / Help
,
1

I went to take a look at the ExUnit.DocTest documentation and the examples provided…

The doctest macro loops through all functions and macros defined in MyModule, parsing their documentation in search of code examples.

A very basic example is:

iex> 1 + 1
2

Expressions on multiple lines are also supported:

iex> Enum.map([1, 2, 3], fn x ->
...>  x * 2
...> end)
[2, 4, 6]

…do not resemble the unit testing implementation in the mix stub file that examples the format for unit testing.

defmodule ModuleNameTest do
    use ExUnit.Case
    doctest ModuleName

    test "greets the world" do
        assert ModuleName.hello() == :world
    end
end

Rather the emphasis is ostensibly on multi-line syntax, and different types of doc testing.

What role does doctest play in mix unit testing?

Does the default file utilise doctest , or is the macro simply present in the starter test file in case of eventual use of doctest tests?

Calling doctest(Module) will generate tests for all doctests found in the module.

It is not clear to me what format these are to take inside the module in order to be identifiable.

2

I would argue really none.

Doctests test the code examples in the documentation and I would suggest not thinking of them in anyway beyond that or as being related to unit testing. As such they’re relatively simple in form and assumptions are made about those examples such that the macro can automatically generate a test for them.

As for mix stub file. It’s really conflating two things. The doctest ModuleName is just testing the examples and the more extensive unit tests are in the test "greets the world" do [...] lines (and presumably tests thereafter. For a brief example this is fine, but I can see where it could confuse matters some, too.

I actually don’t mix the unit and doc tests in my own tests. I separate out the doctest into its own testing module. That way I can run the documentation testing or run the unit tests alone depending on what I want to focus on.

3 Likes
3

Ah thanks a lot, that clears things up.

I am interpreting that doctest validates the tests that I put in @doc for example?

4

To answer this, @doc lines (or perhaps just any comment out line? I’ve never tried anything else) that start with iex> or ...>. The latter is how you do a line continuation. The assertion appears on its own line directly after the last prompt. Basically, you are testing exactly as it would work in an iex session.

@doc """
## Examples

    iex> 1 + 1
    2
"""

As per the docs, you should also indent examples four spaces. Again, I’m not sure if this is strictly required as I’ve never tried any other way.

1 Like
5

Think of it less as “the tests I put in @doc” and more like “it tests that the examples I have in my docs run properly”. The goal is to help you catch changes to your code that invalidate your doc examples, not replace your unit tests with doc tests.

6 Likes
6

This got me thinking if there is a flag or tag that can be used to exclude doctests (none that I can find yet). I feel like putting them in their own file breaks the locality of testable examples as now you have to go elsewhere to see read parts of the doc. Also, how does this affect ex_docs?

7

Yeah, unit testing for development doc testing for illustration.

Thanks again.

8

I believe there is.

I’ve mentioned elsewhere that I split my tests into three kinds of testing: unit, integration, and doc tests; each kind of test exists in testing modules which don’t mix kinds of test. This allows me to do something like: @moduletag :unit. Here’s an example of a doctest module:

defmodule DoctestsTest do
  use AuthenticationTestCase, async: true

  @moduletag :doctest
  @moduletag :capture_log

  doctest MscmpSystAuthn
end

I then use the --only flag with mix test, for example mix test --only doctest. This will run the doctests and exclude the unit and integration. When I run mix test --only integration, I get only the integration tests and not the unit or doctests. I have to imagine that the other tag oriented mix test options also work as expected.

However, I have also found that explicitly tagging the doctests like I am isn’t necessary. Even when I don’t add the @moduletag :doctest module attribute, the tag is still respected when including or excluding the doctests as though I added it. That tagging must be happening behind the scenes (or something to that effect). I would expect that this probably would work for your scenario of mixed unit/doctest files if you tagged everything else individually; a @moduletag might override the doctest identification (or not, I’ve not tried it)… but it gives hint that excluding doctest on the command line without any additional tagging might be possible.

2 Likes
9

OHHHHHHHHHHHHHHHHH I completely misunderstood! Yes, this makes a lot of sense and I quite like this idea. I thought you were saying you moved the actual documentation to a different module from their functions which is what raised my eyebrow :sweat_smile:

2 Likes
10

Hehe… I actually, do in fact do that… but that’s a different topic I believe

I misread your comment. I don’t split docs away from the thing that they document; the API surface of the module is split from the implementation… and the docs follow the API surface functions.

1 Like
11

Would splitting effect the doc format (i.e. when the htm doc is generated)?

12

No, not really; but I think I’m not understanding the question… especially because you were replying to me at the same time I was updating my comment to correct a different misunderstanding of mine.

If I do understand the question, the document generation works as normal. I just run plain old mix docs to get the documentation out. I have done a fair amount with ExDoc configuration in mix.exs, but that’s about it.

13

bahahaha, I read your response while out walking my dog and now that I’m back it’s crossed out so no need for an off-topic :smiley:

Just for clarity, are you talking along the lines of defdelegates and the like here?

14

I wanted to ask if taking the examples out of the @doc above the method would need remove change their position in the htm.

The question is moot now as you do not do that. (put the docs in a different file)

Also I was trying to figure out how to use the crossout a few days ago so thanks for that as well.

1 Like
15

Yeah, a lot of defdelegate, though if the call is to, say a GenServer, then I wrap the GenServer call directly. This is an example of an “API” module using both call styles (and their docs :slight_smile: ):

Summary

musebms/app_server/components/system/mscmp_syst_settings/lib/api/mscmp_syst_settings.ex at next_version · MuseSystems/musebms · GitHub

1 Like
16

Ya makes sense, I’m a big fan of defdelegate myself. I see it as one of the keys to not getting stuck in cyclical dependency hell (also of just nicer code in general).

Thanks for the link to that repo! It’s good reading material for someone like me who works alone these days :slight_smile:

1 Like