Mneme - Snapshot testing tool for the busy programmer

I’d like to announce Mneme, a snapshot testing tool I’ve been working on.

I’ll keep this post short for now. This has been a lot of fun to work on and I hope that others find it useful! Please feel free to post any comments/feedback here, or issues on GitHub.

36 Likes

This looks great! Thanks for the library.

The GitHub link doesn’t seem to work though?

Fixed!

@zachallaun this looks great! I’ve tried it and so far this looks promising. I have one question though:
the interactive dots during the test suite run only appear at once after everything is finished. On bigger codebases this would be a no-go, since the immediate test feedback is really important here.
Is there a way to bring this “progress-bar” feeling back?

Also nice that you link to https://insta.rs/ and GitHub - janestreet/ppx_expect: Cram like framework for OCaml , the ideas behind those libraries are quite cool!

More links, if anyone is interested:

3 Likes

Thank you so much for trying it out and for the question!

To make sure I better understand, are the dots appearing all at the end even when there are no prompts from Mneme? If so, that’s a bug and can/should be fixed. It’s currently expected that nothing will appear during prompting, however, even though async tests are continuing to run in the background — I’m currently suppressing output to prevent garbled terminals during prompting.

One possible solution I explored was a “raw mode” TTY, so that Mneme would have full control over the terminal and get keys on press instead of on-enter. This turns out to be a really tricky problem in Elixir/Erlang, however, though some changes coming with OTP 26 might make it easier in the future.

This article and discussion is what got me working on this. :slight_smile: I’ll add an “additional reading” section to the docs with some of this. Thanks for sharing here!

1 Like

I mean the test run without any prompts, it looks like the process is hanging and then all the dots appear at once.

Raw TTY sounds indeed like a challenging problem, especially with async tests running. I currently do not a good idea how to approach it. Should I open an issue on the repo with a reproducible example?

If you have one, that would be great! Thank you!

Hey @zachallaun , here is a small demo:

I run tests in IEx, so I’ve adjusted mneme to restart on Mneme.start() fn call.

Thanks so much for the demo! I’ll see what can be done.

1 Like

@mindreframer I’ve just published 0.1.1, which pretty dramatically reduces the performance gap between auto_assert and assert. If you are able to give it a try and let me know what issues persist, I’d greatly appreciate it.

This was most definitely worth pursuing! Your demo made is super clear just how slow auto_assert was compared to normal asserts.

My intuition about compile-time vs. runtime costs was wrong in the context of .exs, since all compile-time costs are incurred every time you run the test. By dramatically simplifying the code that the auto_assert macro produces, an unscientific benchmark of running a test with 100 generated auto-asserts goes from taking ~3 seconds to taking 0.2s (compared to 0.05s for normal asserts).

1 Like

@zachallaun WOW! I’ve tried it and now it feels buttery smooth! “Definitely worth pursuing”, as you’ve said. I think it’s ready to be tried out on real codebases.

Would you be open to a PR, that restarts Mneme.Server on each Mneme.start() call? I run my tests interactively in IEx and currently the call the Mneme.start() from the test_helper.exs fails on 2nd run and restarts the whole IEx process. This is the commit that implements it: Feat: allow Mneme to be started repeatedly in a persistent process · mindreframer/mneme@4c17f03 · GitHub
Also there are probably cleaner ways to do it :slight_smile:

Definitely open to it or something like it, possibly behind an option to Mneme.start(). I need to put a bit more thought into how Mneme should be started anyways — right now it’s sort of just “simplest thing that works”.

@mindreframer I just pushed a change to main that adds a :restart option. When you have a moment, would you mind trying it out and seeing if it does the trick?

# test_helper.exs
ExUnit.start()
Mneme.start(restart: true)
1 Like

@zachallaun P-E-R-F-E-C-T! I’ve tried it and it “JUST WORKS”.
Thanks for this package, looking forward to work with it on some real code.

You’ve made my day! Thanks for the awesome “customer support” experience, it was really fun! I hope you’ll enjoy using mnene as much as I will :grin:
Have a nice day!

1 Like

Thanks so much for the kind words! Please let me know if you have any more feedback as you use it. In particular, if you ever find that the patterns being generated aren’t useful, please open an issue!

Edit: Released v0.1.2 to include that :restart option. :slightly_smiling_face:

I just wanted to let you know that I’ve just tried using the tool and it’s awesome! Love it so far!!

1 Like

Thank you so much for the kind words, @tcoopman!

I’m hoping to record a slightly longer screencast soon showing off some workflows that this enables. One I’ve been really enjoying is a kind of variant of REPL-driven development, where I have a pane open with a set of example calls to a function I’m working on using the @mneme action: :accept option to auto-accept changes. So as I work on it, I can just use a single keybinding and see all the calls update with whatever it’s currently returning.

@mneme action: :accept
test "wip" do
  auto_assert some_function(args1...)

  auto_assert some_function(args2...)

  auto_assert some_function(args3...)

  auto_assert some_function(args4...)
end

Then once everything is as I expect, I remove the @mneme action: :accept and refactor into more organized/useful tests.

Edit: I actually just ran into some difficulty with this because Mneme selects the simplest pattern by default, so if you are doing the above with a function that returns a map, it’ll generate a %{} pattern, which doesn’t give you much information. I’ve just pushed a new :default_pattern option that can be used to specify which pattern should be selected:

@mneme action: :accept, default_pattern: :last
test "wip" do
  ...
end
1 Like

:star_struck: this. is. amazing.

1 Like

Thanks to everyone who’s tried this out so far. There have been several patch releases since I initially posted. Currently on v0.1.5, but all changes since v0.1.0 summarized below:

Enhancements

  • Dramatically reduce the performance gap between auto_assert and ExUnit’s assert.

  • Add a :restart option to Mneme.start/1 to restart Mneme if called multiple times.

  • Add a :default_pattern configuration option for auto-assertions which controls the pattern that should be selected by default when prompted.

  • More consistent handling of charlists: lists of integers will now generate themselves as a pattern as well as a charlist if the list is ASCII printable (#6).

Fixes

  • When converting an auto-assertion to an ExUnit assertion, select the identical pattern when the :default_pattern is :infer (set by default).

  • Fix a bug that could cause auto_assert expressions to revert to the previous value when using Mneme.start(restart: true) (#7).

2 Likes