While refactoring my project code its output is routinely checked manually against results generated with the current stable version. I’d like to automate this and my instinct is to add the stable version’s codebase to the new codebase’s lib folder to do so but that doesn’t strike me as a best practice. Any ideas/recommendations?
Perhaps I’m missing some context here, but if you need to ensure that your refactored code ends up with the same output as the stable version you just need to write automated tests that expect exactly that output.
If the implementation in the codebase changes, but the tests do not, this should provide you the assurances you need, right?
Or is this output of some format that is not easy to assert against in an automated test?
So, like:
- Record the inputs used by the stable version along with the resulting output
- Create a test for the refactored version based on the same inputs
- Verify that the results generated by the refactored version exactly matches what was produced by the stable version
Am I on the right track?
If it is feasible, ideally you’d create the automated tests prior to writing any new code. Those are then in place to ensure changing the implementation going forward does not impact the output.
You can probably achieve the desired result with roughly the steps you mention if you already have a lot of new code. However, you might consider checking out a new branch off of stable, writing some automated output tests, and then merging into your new version branch.
You can then always run your automated test suite as you make changes in the future to ensure nothing has broken.
1 Like
No, not really:
- Make tests that assert on the right results from the current (non-refactored) code that supposedly is the baseline and works correctly.
- Make MANY such tests, time budget allowing.
- Introduce the refactoring.
- Re-run the same tests.
- If there’s a different output (and a test fails) then it means your refactoring introduced a bug.
- You should be using a version control system like GIT so you have what to revert to if you can’t fix the bug (so the older version there should be the original code).
1 Like
Thank you both for your insights! The challenge here is that the algorithm consumes fresh data every few seconds from an external API (think Stock Ticker) which means that the output is always changing. The only way I’ve come up with to make sure that a refactor behaves as intended is to feed the same input to both versions and then compare their outputs.
Based on your recommendations, though, it seems that going the Property Testing route is the much better approach since it would guarantee that each piece of code is doing exactly what it’s supposed to regardless of the inputs being consumed. Does that make sense?
Interesting idea, if I have this right you are looking at something similar to snapshot testing in react/jest? (https://jestjs.io/docs/en/snapshot-testing) Not seen anything that does this in the elixir world but yeah it could be an interesting concept to explore, I’m inclined to think the other suggestions are better approaches for business logic still it could work in some situations.
1 Like
Then you should capture a good amount of that ever-incoming data and encode it as static and non-changing input to your unit tests.
Advanced topic for later: property-based testing. But don’t think about it now. Just capture static data and use it in your tests. Add the maximum amount of assertions that would make you comfortable.
1 Like
Never came across that but yeah that is a pretty similar concept. I guess in theory it can be applied to many other pure functions but ultimately it is usually a pretty similar use case. I’ll be sure to check it out 
Also check bypass. Used it a few times, it’s very helpful to mock some services.
Although I suppose we should also be using mox nowadays.
1 Like
Both mox and bypass are excellent, I’ve generally used them for slightly different use cases most of the time.
