Questions about Property Testing / Stream Data

Property-based testing

(Note that I call it property-based testing, rather than property testing, because it’s not the property you test, but you base your testing on properties. Moreover, that is the term we used when we invented this way of testing and I like to stick to that origin. Call it whatever you like.)

As @josevalim states in this post, there is all reason to assume that eventually you write better software if you use property-based testing. With many years experience, I can state that it has changed my way of developing software. If I cannot think of, at least, a simple property that should hold for my code, then I cannot even start writing that code.

We also experienced that it is harder for people to think in terms of general statements of their programs than to think in terms of examples. So my advise is to start with that example and then to say, why 20, why not 40 or 80? Generalize to arbitrary 20-faults

     let n <- nat() do
          n * 20
     end

and use these random values as base of your tests. Start simple and extend.

That’s nice to know! I do like “property-based testing” because it is a direct contrast to “example-based testing”, although I am guilty of using both terms interchangeably. We will make sure to use “property-based testing” on all documentation from now on.

Thanks for sharing these criteria.

I assume this post is at least in part a reaction to the latest Elixir Fountain episode. When I was listening to that episode, I found myself wondering, “Why include a property-based testing library, but not a JSON parser?” (JSON parser is just a random example of a frequent need I find that I have.)

I don’t tell you this to question your judgement. I’m saying that hearing your reasoning helps people like me work out how such decisions are made. It also helps us if we’re going to lobby for the inclusion of some item in the future, because we know a little about what it would take to convince you.

That said, I do have a question related this addition. Does this new library indicate a blessing of the core team to embrace the check all syntax “trick” for handling variable arguments? I call it a trick, because I had to play around with the code to figure out that all serves as a non-existent function call in order to “trick” Elixir into building the AST of property assignments. This surprised me a little.

I had assumed Elixir’s preferred construct for grouping expressions was a block. I guess I got this impression primarily from Kernel.SpecialForms.try/1. Although I guess Kernel.SpecialForms.with/1 is closer to this case. Anyway, I guess I had expected we would solve a problem like this with code closer to:

given do
  left <- string(),
  right <- string()
after  # I wanted `then` but I guess it's not legal:  https://hexdocs.pm/elixir/syntax-reference.html#content
  assert String.contains?(left <> right, left)
  assert String.contains?(left <> right, right)
end

Anyway, this is just a curiosity of mine. Thanks for your time!

I love this. While people may come for the speed/scalability of Elixir, they may well stay because it not only makes easy, but actually encourages you into using decent coding principles that help you build reliable resilient systems

I know I’ve said it a million times before but this aspect of Elixir is sooooo appealing to me!

I think for this case I am in the camp of “the ends justify the means”. The reason is that if property tests are going to be pervasive, the syntax needs to be beautiful and accessible. I really like that syntax; it’s very simple and direct. The generators make sense as arguments for me.

I think explaining this feature as if it were a special form is just fine,

But I definitely appreciate the thought that these kinds of hacks should be used with extreme care.

Thanks @JEG2, those are all excellent points.

I don’t believe it is a problem to question my judgement. There is a thought process and you can disagree with the outcome of the thought process or even with the thought process itself.

I would, however, be unhappy if the main assumption is that there is little to no thought process. Part of this unhappiness is with myself, because it means I did not communicate as well as I should have, but also partially with the assumption in itself, since at this point I hope we have shown that we do better than that.

I would like to add a small footnote related to when we add things to Elixir: we also need to integrate the feature with the language “naturally”. For example, the Decimal package is extremely important for Elixir developers but we are unable to integrate it in the language in any meaningful way. If the is_number/1 guard cannot return true for decimals, adding it to the language will rather make things inconsistent. That’s why the focus on stream data is important, because that’s our integration point.

JSON, CSV and similar are necessities rather than concepts that are important for Elixir developers. URI is another necessity but one we needed in Elixir/Mix.

I am a quite partial to this trick since I mentioned it in the first ElixirConf, even before 1.0, to support constructs such as stream for and parallel for in the language. The goal of check all is to mirror with and for and that’s the best we can do without variable args.

I am not a big fan of the do/end approach because, although we use do/end blocks for grouping expressions, those expressions are typically executed sequentially unless there is a non local return. Generators would look fine inside do/end but filters would look too loose. For example:

given do
  left <- string()
  left != ""
  right <- string() 
  if right != left do
    ...
  end
after
  ...
end

AFAIK there is also nowhere in Elixir where we traverse the expressions in a block and rewrite them, which this approach would effectively require. But at the end of the day I concede most of it boils down to personal taste.

I think you’ve explained it very well why stream_data is set to be included in Elixir. I think that including something like this in Elixir might discourage developers from trying different approaches (which is bad), but this disadvantage is dwarfed by the fact that you can use it to test Elixir. My only worry is that when stream_data is finally officially shipped with Elixir it might start moving “too slow”, but if anyone disagrees, they can just fork stream_data and publish their own version, so no harm done.

For example, I would like to play with Hypothesis’ bytestream-fuzzing approach and maybe release something based on that, but I support the inclusion of stream_data if it is useful to test the Elixir codebase.

Yeah this is the one part of stream_data that I’d change. I use stream_data a lot but honestly it would flow a lot better for me if I could do things like:

check do
  integer_stream <- integers() # grab an integer
  integer_stream >= 0 # skip on a boolean false
  someFuncCall(integer_stream) # Fail if this exceptions or returns false, continue otherwise
  string_stream <- string() # grab a string
  anotherFuncCall(integer_stream, string_stream) = Integer.to_string(integer_stream) <> string_stream) # fail if no 
match
  ...etc...other...tests...
end

Could even prefix things with assert and such too. But such multi-stage building would make some of my tests more clear as well.

Plus no hack of multi-arguments (I am really really anti-dynamic-arguments apparently, it just does not feel right on the BEAM at all, blocks are better for such things, like they would be for for and with too, but that is not this issue ^.^

But yes, stream_data is awesome other than that one ugly syntax quirk, I use it quite a bit now instead of the others.
Main ‘feature’ it really needs in my opinion is state testing and reduction now like some others have, that way we can test full-on processes instead of just simple functions.

It ‘could’ if the Decimal module was not a weird Struct type and instead was a tagged tuple, those can be tested entirely in-guards.

Yeah I still really really hate variable args. If you need a variable args in something then pass in a list or tuple, just wrapping a macro around a pre-call or baking it into the language makes it look really out of place and unnatural. See my comp for replacement for example here:

  iex> comp do
  ...>   x <- list [1, 2, 3]
  ...>   x
  ...> end
  [1, 2, 3]
  iex> comp do
  ...>   x <- list [1, 2, 3]
  ...>   x * 2
  ...> end
  [2, 4, 6]
  iex> l = [1, 2, 3]
  iex> comp do
  ...>   x <- list [1, 2, 3]
  ...>   y <- list l
  ...>   x * y
  ...> end
  [1, 2, 3, 2, 4, 6, 3, 6, 9]

It expands to code that is a lot more efficient than for as well in benchmarks (Elixir’s for is 1.34x slower on average) while supporting more types and a combination of types (combine list and binary comprehensions for example), but the main point is I find it a lot more readable as it is in a block instead of having comma-droppings all over the place with very weird alignment performed (the ‘type decorators’ of list and map and such could be left out, but having those let me generates more efficient code, you could always go with a generic Access though too for a slight speed hit, about on par with Elixir’s for then, but still with a more readable and extendable syntax).

But stream_data’s check all has a similar issue, it should not be a variable-arity-like call but should be a block, blocks are awesome and more readable (they would be awesome in erlang, even OCaml has blocks! ^.^).

Yeah that style I would not like especially… The expressions and generators should be mixed, like how I do with my comp as it seems to ‘flow’ down the block properly in the right order instead of relying on a variable-argument expression do the same (when you should not normally rely on the order of argument expansion).

Eh, just my normal bits about variable-argument functions should NOT exist, even conceptually, on the BEAM (use lists). ^.^;

But still, love that stream_data is being added, hopefully it finishes up state testing features that other property testing frameworks have (never know, it could find bugs in one of the Task or Actor’s or so ^.^).

You do pretty much the same in ExSpirit (defrule) and in the Expression Template-based rewrite will add variable arguments for pretty much everything (alt([p1, p2, p3]) to alt(p1, p2, p3) and etc.). What’s the difference here?

They are mixed already.. What’s the difference?

I still quite prefer it with lists, in my own code even when testing the new format I’d still use lists, I just know that normal PEG parsers would not usually take a list so it feels unnatural in those cases. ^.^

Currently check all is not mixed, the generators and filters are pre-defined then there is a body block. In some cases it can make sense to mix them like if you only need one generator sometimes unless certain conditions are met that warrant the next, that way it saves processing of always acquiring the second even when not always used.