I have been interested in property based testing for a while and have started experimenting with ExCheck.
Using it to test a server has not been very successful (probably just not a good match with the problem) so I was wondering who has been using it and why?
Do you use property tests and unit tests or have property tests replaced some other part of your test suit.
Did you turn to property testing because of a problem in the old way you tested or was it just introduced as a “nice to have” or through an experiment.
Do you use ExCheck? it looks to me like this has the majority mind share, but EQC looks also viable. others?
FWIW, I have been experimenting with ways to turn unit tests into property tests.
It’s still very much a work in progress, and probably the best direction would be to have it run proper underneath rather than my naive implementation. I’m still not sure if I haven’t re-invented dialyzer by
using assertions rather than typespecs.
On my last elixir project I used a combination of unit tests and property tests. The unit tests were developed as a way to design apis and interactions between apis. They definitely weren’t exhaustive of all inputs. In order to test against a larger set of inputs we used property tests.
At one point we needed to generate unique nicknames for people based on their full names. It was actually pretty easy to determine the invariants (meaning properties) for that problem. In fact the generator was harder to create then the tests themselves. Those tests managed to catch a few edge cases in our implementation which gave us a lot more confidence in our solution (especially when it came to optimizing and refactoring).
This seems silly to say but the biggest “trick” to property tests was learning how to work out the properties themselves. In some cases that can be pretty hard to do. But I found that it was really useful because it forced me to think through and reconsider some of my early design choices. Composing generators is a skill and took me some time to get used to. But, like anything, it gets easier the more you do it.
I’m definitely not an expert at property testing; I feel like I’m just getting started. But based on my results so far its definitely a skill worth cultivating. The “Little Elixir and OTP guidebook” has a good overview of property based testing. The original QuickCheck papers are a fun read as well.
This seems silly to say but the biggest “trick” to property tests was learning how to work out the properties themselves.
It’s not silly at all, by far and away the hardest part of property testing is coming up with invariants. If I see that stupid silly sorted list example one more time…
I’ll certainly take a look at the little elixir otp guidebook if thats recommended. I think I am still gaining the tricks to really make property based testing work.