Regression testing

Say I have a decoder function f/1. I’d like to check compatibility between several versions using property testing. Something like

# for all x in the domain
assert f_old(x) == f_new(x)

Where f_old is the function f/1 from a previous commit and f_new is the function f/1 from the working copy.

Is there any out of the box solution in Elixir? I was thinking of something like:

  • Start the application (library) in the working dir

  • Clone the repo somewhere else

  • Compile and start the old application (library)

  • Generate test data from the new application (library)

  • Somehow communicate with the new application to run get the result of the old function on the new data

  • Compare to the new result

This avoid the need to keep a corpus of data, and allows me to use a new test data generator to feed the old function. Maybe I’m overcomplicating this and there is a much simpler way (apart for the simplest one of copying the source code of the old version). I have no concrete use case for this yet (I might have one in the future), so I’m mainly asking because it sounds cool.

This idea could be extended for non deterministic functions as long as you have an invariant. For example:

# test if the old nondeterministic encoder
# compatible to the new decoder after
# the optimizations
assert data |> encode_old |> decode_new == data
1 Like