Mneme - Snapshot testing tool for the busy programmer

Getting closer to the next sizable update to Mneme and wanted to share! After noticing that it can sometimes be difficult to visually parse what changed when an auto-generated value updates, I started working on semantic diffing, using the amazing Difftastic as reference and inspiration.

This has proved more of a rabbit hole than I initially thought, but is still relatively easy thanks to Elixir’s general awesomeness. Today I reached the point where my semantic diff development branch can start using semantic diffs to work on semantic diffs!

For instance, here’s what an update might look like using the latest release:

Here’s the same update using semantic diffs:

image

There’s still a ways to go before this lands, but I’m hoping that being able to use semantic diffs in development will help expose edge-cases/etc. before I ship the feature.

8 Likes

Semantic diffing is now in main. If anyone is interested in trying it out, please add :mneme as a git dependency and enable semantic diffs in your test.exs configuration:

# mix.exs
defp deps do
  [
    {:mneme, github: "zachallaun/mneme", only: :test}
  ]
end

# config/test.exs
config :mneme, defaults: [diff: :semantic]

If you run into any errors, please let me know by opening an issue on GitHub!

3 Likes

v0.2 released

Note: Mneme now requires Elixir v1.14 or later.

defp deps do
  [
    {:mneme, "~> 0.2.1", only: :test}
  ]
end

Semantic diffs

In addition to some quality-of-life changes when things go wrong, the big feature that v0.2 brings is a custom diff engine that produces dramatically more readable diffs when auto-assertions are updated. I’m really excited about this and it’s made Mneme much more pleasant to use, at least for me.

I expect there to be some formatting bugs – please report them on GitHub if you encounter anything weird! If you desire the old behavior, you can configure Mneme to continue using textual diffs as before.

Example 1. New assertions highlight only the new value

Text

Semantic

Example 2. Changed assertions highlight deleted/added nodes

Text

Semantic

Example 3. Slightly changed strings call out the changed portion

Text

Semantic

Future work

  • Diff quality: These diffs could still be improved. For instance, example 2 highlighted a suboptimal set of square brackets.
  • Performance: For large expressions, semantic diffs will take too long and it will fall back to text diffs. I’m sure there’s still some low-hanging fruit for improving performance, however.
  • Formatting: I’d like to add side-by-side formatting to decrease the vertical real-estate needed. I’d also like to explore replacing very large, unchanged nodes with ... or something in the same way that inspect does at a certain point.
6 Likes

I’ve released a new version (v0.2.2) that disables an optimization that was causing a poor diff in example 2 above, increasing the likelihood that subtrees would become “misaligned”.

v0.2.1

v0.2.2

The new diff does a much better job of reflecting the actual change: the nil is being replaced with a new subtree, and the second subtree is being modified.

5 Likes

I’ve released v0.2.3 which includes fixes for a few bugs.


I’d also like to mention a proposal I just posted on GitHub for the introduction of mix mneme to run tests with Mneme prompts, which would be favored over the current behavior that hijacks mix test. If you have any thoughts/feelings about this, I’d invite you to read the proposal and react/respond on GitHub. :slightly_smiling_face:

1 Like

v0.2.6 released

defp deps do
  [
    {:mneme, "~> 0.2.6", only: :test}
  ]
end

Changes

In addition to a number of bug fixes, this release updated (and hopefully improved) the formatting for semantic diffs, including the addition of side-by-side diffs if terminal width allows. (This can be disabled with a config option if you prefer stacked diffs.)

An option to “skip” a changed assertion has also been added. This will pass the test, but fail the test run at the end. This can be useful if a test contains multiple auto-assertions, as rejecting the first one would immediately fail that test and prevent the later ones from running.

New demo:

4 Likes

v0.3.0-rc.0 released

I’m hoping to cut a full release soon, but would like to give folks a few days to try things out and report any bugs that I missed, so I’ve pushed this release as an RC.

defp deps do
  [
    {:mneme, "~> 0.3.0-rc.0", only: :test}
  ]
end

Breaking change

auto_assert will now use <- when comparing against falsy values instead of ==, and support for == has been removed.

# old
auto_assert Function.identity(nil) == nil

# new
auto_assert nil <- Function.identity(nil)

When I started on Mneme, I wanted the behavior of auto_assert to exactly match that of assert, including the expectation of a truthy value. Consider the following ExUnit assertions:

# this will fail
assert false = Function.identity(false)

# this would succeed
assert Function.identity(false) == false

# this might succeed or fail
assert some_predicate?()

assert has to expect a truthy value to support that third case, but that case doesn’t make sense for auto_assert. We’re always trying to construct a match pattern, and we use the presence of <- in an AST to disambiguate between a new assertion and one that is being (potentially) updated.

All in all, supporting == comparisons after auto_assert complicated the code and didn’t (in my opinion) add significant value, so I’ve decided to deviate from ExUnit in this particular way and to allow matches on falsy values to succeed.

New stuff

In addition to a number of fixes, this release is largely about introducing auto_assert_raise, auto_assert_receive, and auto_assert_received.

Example: auto_assert_raise

test "raises when passed invalid arguments" do
  auto_assert_raise fn -> some_call!(:invalid) end
end

Running the test gives you two auto-generated options, one with and one without a message:

test "raises when passed invalid arguments" do
  auto_assert_raise ArgumentError, fn -> some_call!(:invalid) end

  # or

  auto_assert_raise ArgumentError, "expected message", fn -> some_call!(:invalid) end
end

Example: auto_assert_receive / auto_assert_received

Example test case:

test "current process should receive an ack" do
  ref = send_request!()

  auto_assert_receive()
end

Running the test will prompt you with options for each message received by the current process during the timeout window (default 100ms, same as ExUnit’s assert_receive):

test "current process should receive an ack" do
  ref = send_request!()

  auto_assert_receive {:ack, ^ref}
end

As with ExUnit, auto_assert_received is the same as auto_assert_receive with a timeout of 0.


As always, any feedback is greatly appreciated!

3 Likes

Hi, did you consider adding auto_refute?

Yes, that was my original plan, but I decided against it because I feel that Mneme doesn’t provide value when you’re simply asserting against a boolean expression. In that case, you should just be using assert or refute directly.

I also think that it would be more confusing to see your auto_assert being converted to an auto_refute because it began returning a falsy value, instead of auto_assert false <- whatever(). Unifying on auto_assert keeps the return value obvious and explicit.

3 Likes

v0.3.0 is now available

This is the biggest release yet, including new auto-assertions (mentioned above), quality-of-life improvements, and numerous bug fixes. A couple of additional things made it into the release since the first RC:

  • One-line summary is now printed at the end of the test run.
  • :force_update option is now available, forcing patterns to regenerate even when they would succeed. This is especially useful when you’re working on something using maps or structs, since patterns would not re-generate when a key is added. (Thanks to @mindreframer for suggesting this feature!)

Please see the changelog linked above for the full list of changes!

3 Likes

v0.3.3 is now available

The last couple of releases have been smaller fixes and improvements to multi-line string handling, but this release of v0.3.3 has a pretty significant quality-of-life change that I thought it was worth mentioning here that much improves the default pattern selection for maps and structs.

Consider the following new auto-assertion:

user = %{first_name: "Zach", last_name: "Allaun"}

auto_assert put_role(user, :moderator)

When this runs, it will generate two patterns that you can select from:

auto_assert %{} <- put_role(user, :moderator)

auto_assert %{first_name: "Zach", last_name: "Allaun", role: :moderator} <-
              put_role(user, :moderator)

Now let’s say you select the second one, but you modify the assertion to only assert on the role:

auto_assert %{role: :moderator} <- put_role(user, :moderator)

Previously, if the result of put_role changed (say, it starts returning roles in a list), Mneme would only generate the “empty” and “full” patterns again, and you’d have to re-edit it down. As of v0.3.3, Mneme will also generate a pattern for the subset of keys you’re asserting on, and select it by default:

This fixes one of my personal pet-peeves. One of the most important tenants guiding Mneme is to suggest the assertions you’d write yourself, and this particular case came up surprisingly often in my own usage.

4 Likes

I wrote a brief Mix.install-based “tour” of Mneme for folks who might want to try it out without having to add anything to a project. The code can be found here. To give it a try, just download and run using elixir! (requires Elixir 1.14+)

curl -o tour_mneme.exs https://raw.githubusercontent.com/zachallaun/mneme/main/examples/tour_mneme.exs
elixir tour_mneme.exs
3 Likes

Really nice idea to ease the testing part! I will try it in the next days.

Thank you for your contribution @zachallaun !

v0.3.4 is now available

This release should be very small in terms of visible changes, but adds a number of quality-of-life enhancements and fixes numerous edge-cases related to pattern generation (thanks to stream_data!).

Mneme is now also tested against Elixir 1.14.4 with OTP 26.0. It should be working with Elixir 1.15.0-rc.0 as well, but a small bug is preventing tests from passing, but I’ll hold off on saying it’s compatible for sure :slightly_smiling_face:

1 Like

v0.3.5 is now available

This is a small release that primarily consists of internal changes & refactoring, though it also fixes a couple of minor bugs with Elixir 1.15+, including the use of multi-character sigils.

Mneme is now tested against Elixir 1.15.1 and Erlang/OTP 26.0.2, so you should feel confident if using the latest and greatest!

3 Likes

v0.4.0 is now available

This release introduces one breaking change: config :mneme, ... is now replaced by passing options directly to Mneme.start/1 when you wish to apply the options to the entire test run. If you weren’t using Mneme’s application config previously, you do not need to change anything. If you were, please see the changelog above for an example.


While I’m at it, here’s a funny little snippet that shows how Mneme can be used to (almost) delete itself, replacing all existing auto_assert* with regular assert*:

Mneme.start(
  target: :ex_unit,
  force_update: true,
  action: :accept
)
4 Likes

v0.4.3 is now available

There’s been a small flurry of bugfixes and qualify-of-life improvements in the last week. Special thanks to @tcoopman for the reports!


A special request: If you find any weird behavior when using Mneme with the Elixir 1.16 release candidates, please file an issue! Some light initial testing suggests that everything should be working as normal, but I’d love to make sure that Mneme doesn’t hold anyone back from upgrading! :slightly_smiling_face:

4 Likes

Special thanks to you @zachallaun!

Mneme is an amazing tool for testing! And you fix bugs so fast :metal:

1 Like

Request for comment!

I’m considering removing empty raw maps from the list of generated patterns. I’d greatly appreciate any comments/feedback on this on GitHub. Thanks!

2 Likes

Great library! I would love to have an interactive credo fixer and compiler warning fixer. Will take a look at the source for Mneme for inspiration.