James_E

James_E

I see that the current ExUnit source code has support for rich failure messages on a small whitelist of “recognized” assertion patterns, namely:

  • assert match?(left, right), assert left = right
  • refute match?(left, right), refute left = right
  • assert left === right, refute left !== right,
  • assert left !== right, refute left === right,
  • assert left == right, refute left != right,
  • assert left != right, refute left == right,
  • assert left >= right, refute left < right,
  • assert left > right, refute left <= right
  • assert left <= right, refute left > right
  • assert left < right, refute left >= right
  • assert left ~= right
  • refute left ~= right

Would there be any interest in a pull request adding support for an additional pattern:

  • refute left and not right (implication, right must be true whenever left is)

This pattern is coming up repeatedly as the most robust, developer-intent-communicating form for some unittests I’m writing:

  property "delete returns a well-formed multiset whenever input is well-formed" do
    check all ms <- t(term(), strict: false),
          value <- term(),
          count <- one_of([non_negative_integer(), constant(:all)]) do
      result = One9.Ms.delete(ms, value, count)

      refute One9.Ms.well_formed?(ms) and not One9.Ms.well_formed?(result)
    end
  end
  property "put 0 copies doesn't corrupt struct" do
    check all multiset <- t(term()), value <- term() do
      result = One9.Multiset.put(multiset, value, 0)

      assert One9.Ms.well_formed?(result.counts)
      assert One9.Multiset.equals?(result, multiset)
      refute One9.Multiset.member?(result, value) and not One9.Multiset.member?(multiset, value)
    end
  end

And having these tests augmented with a little bit of extra verbiage and awesome debug formatting, like the comparison checks currently have, would be a minor boon.

Showing Posts 1 to 8

tfwright

tfwright

Personally, I think the most important value for test cases is simplicity, so I’m definitely skeptical about support for something like this.

FWIW, about the first example specifically I would argue that you should not be writing assertions against test inputs since they are supposed to be known. Granted I’m not familiar with the DSL shown so maybe I’m misunderstanding something but generally I would only expect the second part of that assertion to be a meaningful test since the former would be testing the test itself.

James_E

James_E OP

This is just property-based testing. You declare what type of inputs your functions shouldn’t choke on, and the unittester fuzzes your code for you, then attempts to make minimum viable repros of any failing examples.

Tangent

I do use “old fashioned hardcoded-input” testing liberally within the doctests, and I have a few exemplar edge or obscure prototype cases hard-coded in other unit tests. But having a per se stream of well-formed spaghetti being thrown against the walls of my code is extremely valuable and I use it for the heavy lifting.

  property "union basic correctness" do
    check all multiset1 <- t(term()), multiset2 <- t(term()) do
      result = One9.Multiset.union(multiset1, multiset2)

      check all value <- term() do
        assert One9.Multiset.count_element(result, value) ===
          max(
            One9.Multiset.count_element(multiset1, value),
            One9.Multiset.count_element(multiset2, value)
          )
      end
    end
  end
tfwright

tfwright

I had a feeling that was what I was looking at. Nothing against property-based testing as another tool in the arsenal, although personally I haven’t found need for it as of yet in the types of programs I have tested. But I wouldn’t think ExUnit needs to include special support for it.

But also even property tests, as far as I understand, should be able to control inputs at least in terms of types. If one type of input should raise an error, but another should fail gracefully, you need to be able to write a test that isolates both cases, otherwise the distinction is ipso facto irrelevant and shouldn’t be subject to any test. In your example, if the concern is specifically with “well formed” inputs, then I would argue should be a constraint on what is tested in that example, rather than an assertion on the result.

christhekeele

christhekeele

This seems like a reasonable conversation to start on the core mailing list, if you can’t find a prior discussion.

I think property-based testing and other test design philosophy is orthogonal to this discussion:

  • Elixir has special forms and operators
  • ExUnit tries to provide pretty formatting when recognizing some assertions on the language’s forms
  • Whether or not it can do better by recognizing some boolean operator combinations and formatting them specially is a conversation worth having
  • Regardless of testing practices
James_E

James_E OP

That is correct; they do. The issue arises with situations like those given in my OP.

I’m not entirely sure what you’re getting at, here.

Of course, I can do that (and I do do that, in other tests). But when “A implies B” is exactly what I mean to assert, stuffing some of the conditions for A into the generator headers achieves terseness, at the cost of self-documentingness, which is a very raw deal.

I agree almost entirely—except that I suspect “A implies B” is very unlikely to come up as a natural, relevant, correct thing to test outside of property-based testing (which, I suspect, is what @tfwright has been trying to build an argument for this whole time.)

tfwright

tfwright

Agree on the first (property-based) if there is another use case for this syntax, but not so much on the latter (design philosophy). To shift focus back to the latter, my opinion is precisely that support for this would encourage test design I would consider suboptimal on the principles of simplicity and atomicity. I just think tests should generally avoid testing more than one thing at once, regardless of whether that “other thing” is considered an input or not (although if it’s an input then I think it’s particularly bad).

James_E

James_E OP

Ironically, the tooling that’s already in there works great for the strictly more complicated case, “A implies B, and not-A implies not-B”—since === works for booleans.

I file B as left, A as right, and I get an extremely educational error message when things break down that colors A green and B red if they ever disagree:

  property "well_formed basic correctness" do
    assert One9.Ms.well_formed?(%{})
    assert One9.Ms.well_formed?(%{42 => 1})
    refute One9.Ms.well_formed?(%{43 => 0})
    refute One9.Ms.well_formed?(%{42 => 1, 43 => 0})

    check all ms <- t(term(), strict: false) do
      assert One9.Ms.well_formed?(ms) === (0 not in Map.values(ms))
    end
  end

^That’s what got me on the train of thought of pulling out just the first half for assertions.

fuelen

fuelen

When guards are failing, boolean logic is highlighted with colours. I don’t see any reason why it shouldn’t be supported in ExUnit as well.

As @christhekeele has already suggested, please start a discussion on the core mailing list.

— All posts loaded —

Where Next? Top

Trending in Discussions Top

AstonJ
The obligatory hello world thread! Who are you and where are you from? :stuck_out_tongue:
4663 61218 595
New
cblavier
Hey there, It’s been more than a year since we started using LiveView as our main UI library and building a whole library of UI componen...
515 12931 101
New
raa-ziq
Hi everyone :waving_hand: I wanted to share something I’ve been building — NowBlind, a random chat app designed to help people discover ...
New
sntran
In my third year of college, around 2010, I found Erlang. A few years later, I found Elixir. I liked the ideas at once: many small proces...
New
gtcode
I’ve been tinkering with Lynx (by José) via hold_ledger. I used AI to write the code, a separate specification, the tests, and the Lean p...
New
Ganna
Hi everyone, I have 8 years of production experience with Erlang/OTP and I’m just starting to get to know Elixir. What would you learn f...
New
AstonJ
A remark @Garrison made… ..has already inspired a thread for the technical side of reclaiming our internet, but there is of course anot...
New

Other Trending Topics Top

dominicletz
Hey there, I haven’t seen a list of useful Elixir specific skills for the popular AI tools yet so I’m starting one here with my first. ...
New
mudasobwa
I fully migrated to my own harness from Anthropic/Gemini and I think it’s time to share it. Welcome DSH, the DeepSeek Harness, fully writ...
New
OndrejValenta
(I just needed to vent somewhere and LinkedIn is full of hope, or hype, I’m not sure which exactly) AI dream has many faces, but general...
New
kip
Localize is the next generation localisation library for Elixir. Think of it as ex_cldr version 3.0. The first version will be released ...
New
jimsynz
Beam Bots (or just BB for short) is a framework for building fault-tolerant robotics applications in Elixir using familiar OTP patterns. ...
New
mcass19
ExRatatui lets you cook up rich terminal UIs in Elixir, powered by Rust’s ratatui via Rustler NIFs. Build interactive terminal applicatio...
New

We're in Beta

About us Mission Statement

Options

Thread Display Mode




Thread Preview

Skip Thread Previews