StreamData: data generation and property testing for Elixir

I’ve read the blog post about test.check by the author. It does look more more less equivalent to what Hypothesis does if you squint a little.

Well, a seed is even easier to serialize than the Hypothesis byte streams. With a way to serialize text data, you could (and should) save the examples that falsify the tests, so that they could be tested in a next run. And also do other cool things that depend on having serializeable examples.

I think you hsould shrink the examples as much as possible, and that includes shrinking <<16, 16>> to <<0>>… I’ll open an issue tomorrow.

Well, if including specific examples would complicate the code too much, then merely a high probability might be acceptable, especially if combined with a database of failed examples.

Nice job with the library!

This looks really great @whatyouhide! It seems similar in nature to Dave Thomas’s quixir which I’ve been using for a bit. I’m excited to see if I can make this work with some existing state machine properties that I have. I love that these tools may make their way into elixir itself.

Makes it much easier to test GenServer’s though with this. That is something I like about Quivic’s (sp?!?) QuickCheck, it was able to reduce GenServer and remote process calls so nicely.

I’ve been building model based properties with quixir for a while. It works pretty well. Granted, you have to build pre-condition and post-condition checking yourself and the shrinking is much less optimized for model properties then eqc. Plus you don’t get the PULSE scheduler for inducing and controlling race-conditions which is really the most critical piece IMO.

Yes, I would support this. My PropCheck library (PropCheck - Property based testing for Elixir — PropCheck v1.5.0) uses PropEr to do the major work, but provides Elixir-based APIs, including the state-based stuff for GenServers and the like. For practical purposes, you would like to have this available. Otherwise, property based testing boils down to data structure testing only. PropEr has the disadvantage that it does not support maps (and this structs), is not updated a longer time (read: it is stable ;-)), and has a complicated code basis. Therefore, a fresh alternative looks very promising.

Tthe fundamental part for state-based property testing is data generation (which is quite easy) and a good shrinking strategy (which is way harder). With that available, a state-based approach should be easily implementable on top.

This is in quite contrast to Pulse from Quivic, where they manipulate the Erlang scheduler cleverly to find race conditions. While again based on generation and shrinking, here the infrastructure is much more complicated, which becomes evident, if you read their research papers.

I just published:

which is a post where I describe in detail the inner workings of StreamData for those of you who might be interested.

Great article!

It could benefit from a fleshed out shrinking session in which you explicitly show how the tree is traversed.

Also, some details for people who are curious about the Hypothesis approach:

1. Hypothesis’ approach is very elegant. But you pay for it with more complexity in the core, and making it harder to write generators. As @whatyouhide says, primitive generators must be written in a specific way, and even though the idea is simple, the details are quite involved. EDIT: composing already-built generators is very easy, though.

2. Describing hypothesis as taking a random bytestring and generating examples out of it makes it look way simpler than it is: doing this would require an mapping that can generate an example out of any random byte stream thrown at it. This is not true. Sometimes hypothesis cheats and “generates” the bytes it needs in order to satisfy some data constraints (a list with at least 5 elements, for example; in this case hypothesis might force some bytes to have the value it wants). The corollary is that during shrinking, some byte strrams might generate invalid examples, which must be discarded. These are not major problems, but they must be managed carefully, and that takes more code.

3. Shrinking is not done by only shrinking the byte stream. That would generate invalid data too often, and waste time on impossible shrinks. Hypothesis has a way of tagging some bytes as special, so that they will be shrunk in specific ways. In practice, shrinking has some restrictions. This again makes the core much more complex than just “eat some bytes and generate some objects”.

My undeestanding is that Hypothesis might unlock some shrinks that are hard to unlock when using the Lazy Tree approach, but I don’t know if that’s relevant in practice.

If you ever decide you want to go the byte stream route, I can send you my partial (and probably very buggy - haven’t tested it in ages) port of the Hypothesis core to elixir. The Python code is challenhing to understand in some parts, because it mutates objects like crazy and bends backwards to undo some mutations when it decides to backtrack. An elixir port should definitely return modified “generator state”. That gives you undo for free.

I just tried this example:

property "lists of integers don't contain 42" do
  check all list <- list_of(int()) do
    assert 42 not in list
  end
end

And it shrinked the list down to this, which initially through me off quite a bit until I remembered charlists

list <- list_of(int())
#=> '*'

Maybe there could at least be a warning, because the module does seem to know it’s dealing with integer values and not with charlists.

How could it know? Can elixir distinguish characters from integers? I think it can’t…

And that shrink isn’t actually very impressive. The minimum example should have been the empty list, right?

It can’t know from the list itself, but it might be able from the generator used.
Also the shrink is correct. does not include 42, so the assert is fulfilled.