Doctest documentation vs testing in a mix project

DidactMacros

I went to take a look at the ExUnit.DocTest documentation and the examples provided…

The doctest macro loops through all functions and macros defined in MyModule, parsing their documentation in search of code examples.

A very basic example is:
iex> 1 + 1
2
Expressions on multiple lines are also supported:
iex> Enum.map([1, 2, 3], fn x ->
...>  x * 2
...> end)
[2, 4, 6]

…do not resemble the unit testing implementation in the mix stub file that examples the format for unit testing.

defmodule ModuleNameTest do
    use ExUnit.Case
    doctest ModuleName

    test "greets the world" do
        assert ModuleName.hello() == :world
    end
end

Rather the emphasis is ostensibly on multi-line syntax, and different types of doc testing.

What role does doctest play in mix unit testing?

Does the default file utilise doctest , or is the macro simply present in the starter test file in case of eventual use of doctest tests?

Calling doctest(Module) will generate tests for all doctests found in the module.

It is not clear to me what format these are to take inside the module in order to be identifiable.

15 comments

#learning-elixir #mix-format

0 1954 15

2024-01-17 15:09:15 UTC

Showing Posts 1 to 10

Show Best Posts
Show All (oldest first)
Show All (newest first)

sbuttgereit

I would argue really none.

Doctests test the code examples in the documentation and I would suggest not thinking of them in anyway beyond that or as being related to unit testing. As such they’re relatively simple in form and assumptions are made about those examples such that the macro can automatically generate a test for them.

As for mix stub file. It’s really conflating two things. The doctest ModuleName is just testing the examples and the more extensive unit tests are in the test "greets the world" do [...] lines (and presumably tests thereafter. For a brief example this is fine, but I can see where it could confuse matters some, too.

I actually don’t mix the unit and doc tests in my own tests. I separate out the doctest into its own testing module. That way I can run the documentation testing or run the unit tests alone depending on what I want to focus on.

Post #1

DidactMacros

Ah thanks a lot, that clears things up.

I am interpreting that doctest validates the tests that I put in @doc for example?

Post #2

sodapopcan

To answer this, @doc lines (or perhaps just any comment out line? I’ve never tried anything else) that start with iex> or ...>. The latter is how you do a line continuation. The assertion appears on its own line directly after the last prompt. Basically, you are testing exactly as it would work in an iex session.

@doc """
## Examples

    iex> 1 + 1
    2
"""

As per the docs, you should also indent examples four spaces. Again, I’m not sure if this is strictly required as I’ve never tried any other way.

Post #3

benwilson512

Author of Craft GraphQL APIs in Elixir with Absinthe

Think of it less as “the tests I put in @doc” and more like “it tests that the examples I have in my docs run properly”. The goal is to help you catch changes to your code that invalidate your doc examples, not replace your unit tests with doc tests.

Post #4

sodapopcan

This got me thinking if there is a flag or tag that can be used to exclude doctests (none that I can find yet). I feel like putting them in their own file breaks the locality of testable examples as now you have to go elsewhere to see read parts of the doc. Also, how does this affect ex_docs?

Post #5

DidactMacros

Yeah, unit testing for development doc testing for illustration.

Thanks again.

Post #6

sbuttgereit

I believe there is.

I’ve mentioned elsewhere that I split my tests into three kinds of testing: unit, integration, and doc tests; each kind of test exists in testing modules which don’t mix kinds of test. This allows me to do something like: @moduletag :unit. Here’s an example of a doctest module:

defmodule DoctestsTest do
  use AuthenticationTestCase, async: true

  @moduletag :doctest
  @moduletag :capture_log

  doctest MscmpSystAuthn
end

I then use the --only flag with mix test, for example mix test --only doctest. This will run the doctests and exclude the unit and integration. When I run mix test --only integration, I get only the integration tests and not the unit or doctests. I have to imagine that the other tag oriented mix test options also work as expected.

However, I have also found that explicitly tagging the doctests like I am isn’t necessary. Even when I don’t add the @moduletag :doctest module attribute, the tag is still respected when including or excluding the doctests as though I added it. That tagging must be happening behind the scenes (or something to that effect). I would expect that this probably would work for your scenario of mixed unit/doctest files if you tagged everything else individually; a @moduletag might override the doctest identification (or not, I’ve not tried it)… but it gives hint that excluding doctest on the command line without any additional tagging might be possible.

Post #7

sodapopcan

OHHHHHHHHHHHHHHHHH I completely misunderstood! Yes, this makes a lot of sense and I quite like this idea. I thought you were saying you moved the actual documentation to a different module from their functions which is what raised my eyebrow

Post #8

sbuttgereit

~~Hehe… I actually, do in fact do that… but that’s a different topic I believe~~

I misread your comment. I don’t split docs away from the thing that they document; the API surface of the module is split from the implementation… and the docs follow the API surface functions.

Post #9