Option to skip automatic doctests in livebook?

Chat / Discussions
,
1

I’m currently trying to use livebook to work with the suite of ex_cldr packages, but I’m tripping over the automatic execution of doctests.

E.g. for a minimal ex_cldr backend module definition, I’m seeing this error when the cell executes:

Screenshot 2023-06-07 at 10.38.16
Screenshot 2023-06-07 at 10.38.161200×602 30 KB

Evaluation process terminated - an exception was raised:
    ** (MatchError) no match of right hand side value: []
        lib/livebook/runtime/evaluator/doctests.ex:57: Livebook.Runtime.Evaluator.Doctests.report_doctest_result/2
        lib/livebook/runtime/evaluator/doctests.ex:27: anonymous fn/2 in Livebook.Runtime.Evaluator.Doctests.run/2
        (elixir 1.14.5) lib/enum.ex:975: Enum."-each/2-lists^foreach/1-0-"/2
        lib/livebook/runtime/evaluator/doctests.ex:24: Livebook.Runtime.Evaluator.Doctests.run/2
        lib/livebook/runtime/evaluator.ex:465: Livebook.Runtime.Evaluator.continue_do_evaluate_code/6
        lib/livebook/runtime/evaluator.ex:330: Livebook.Runtime.Evaluator.loop/1
        (stdlib 4.3) proc_lib.erl:240: :proc_lib.init_p_do_apply/3

The problem is that use Cldr in the backend module generates a bunch of code that includes doctests. And usually those doctests would pass, but the test setup does not seem to be replicated 100% when the code is generated in this way.

Would it make sense to add a flag for skipping doctests in cases like these?

2

Hey @petrus-jvrensburg do you have an example livebook I can run that shows this issue? How are tests being run at all?

3

I believe the error arises because the backend configuration you defined does not include the locales that the doctests use. In this case :pl is not configured.

You can chose to omit the docs for a backend module by adding generate_docs: false to the backend configuration and I believe that will mean no docs, and therefore no doc tests would be executed.

Its new to me that Livebook is executing doctests in this way though. It seems a little unexpected - what am I missing?

4

Hey, it can happen when use Mod defines functions with doctests. Livebook should ignore those, we are looking into this : )

5 Likes
5

That’s great to hear @jonatanklosko. They are, I think, first and foremost docs. That they are tests is almost a side affect.

6

That’s great. It would make sense for these to be ignored.

And thanks for all your effort, livebook is really good!

7

FTR fixed in Don't run doctests for generated functions by jonatanklosko · Pull Request #1966 · livebook-dev/livebook · GitHub, will merge when the next Elixir v1.15 rc is out (but the fix also works with v1.14).

4 Likes