This is very, very cool. Mutating the test file interactively is a great way of going over the natural laziness of writing tests in cases that are not good for property testing. There seems to be a limitation in which it only works for strings, right? Why don’t you just convert the value into an Erlang term, store it into a file (or even write it inline)? It’s strictly better than using strings in my opinion.

Now, I might have a use for this. I’m the author of the Makeup syntax highlighting library. It needs some automated tests (badly, it’s > 1000k lines of code with almost no tests), as currently all testing is visual integration testing by examining the output (there is some low hanging fruit to be had from property testing, but that’s limited).

assert_value seems like a good tool for the job: I can just dump some source code snippets somewhere, run assert_value on them, compare the output (at least the tokenization, comparing the HTML output is harder). Ultimately, this is waht I’d like to have:

I’m comparing the visual output for my program, and it happens to be in HTML. I’d like to select some code snippets, have them converted into HTML and shown in some form of HTML canvas somewhere so that I could mark them as correct. The most immediate way I can think of implement this is to have the tests start an HTTP server, start a web browser and then ask the user if the output is correct or not.

For example (ridiculous mockup I’ve done in 5mins):

image.png913×401 13.6 KB

An then repeat for all test cases.

This is awesome - had a similar idea recently about updating tests like this, so glad somebody else made it! Awesome work!

Co-author here. Thank you!

Left argument can be anything with implemented String.Chars protocol (String, Atom, Integer, Float… etc.). In all other cases you have to serialize left argument to string. Right argument should always be a string heredoc.

We use heredoc because we need to modify test source code. Currently it is hard to search/replace values in the code because Elixir’s AST gives us only line number without column offset. With heredocs it is easy to search for """ and replace everything in-between.

We are planning to support all types of arguments in Elixir 1.6. It will have formatter metadata in AST which will help a lot.

Exactly! assert value can test for expected values stored in files. You can even compare content of two files:

assert_value File.read!("input.txt") == File.read!("reference.txt")

It is smart enough to recognize File.read! as right argument and will update contents of a file. With Elixir macros you can easily create test for bunch of files in some directory:

defmodule Support do
  defmacro build_tests do
    filenames =
      Path.expand("data/*.txt", __DIR__)
      |> Path.wildcard

    for filename <- filenames do
      quote do
        test unquote(filename) do
          assert_value File.read!(unquote(filename)) == File.read!(unquote(filename <> ".ref"))
        end
      end
    end

  end
end

defmodule MyTest do
  use ExUnit.Case
  import AssertValue
  import Support

  build_tests()
end

Having something like this you will have a separate test for each file and any failed test will not break other tests. And you can start with just input files without reference files:

~/>mix test
test/my_test.exs:34:"test test/data/foo.txt" assert_value File.read!("/home/examp... failed

-
+foo

Accept new value? [y,n,?] y
.
test/my_test.exs:34:"test test/data/bar.txt" assert_value File.read!("/home/examp... failed

-
+bar

Accept new value? [y,n,?] y
.

Finished in 3.6 seconds
2 tests, 0 failures

Hope this will help.

This reminds me very much the idea of approval tests.

http://approvaltests.com/ and GitHub - approvals/ApprovalTests.Ruby: Approval Tests for Ruby · GitHub

Yes. Actually, I think the idea is very simple: you just generate data once, confirm that it’s the correct result and then check for regressions.

The trick is having a good implementation for the real bottleneck, which is human inspection of the results the first time they’re generated. assert_value seems to provide a good UI.

But there are other possible design decisions that might lead to different (and possibly better) UIs. For example:

• assert_value is synchronous. It stops at each test result that hasn’t been checked. Why not making that async after all test have run? One could gather all test results pending confirmation and show them all at once to the user once the test suite has run. I don’t know if this brings any benefits, just thinking about it.

• Another possible improvement (although a bit specialized) for programs that generate reproducible HTML would be to show the results in a web browser through an embedded Plug app. The user could inspect the result visually instead of arsing it tag by tag. This is what I discussed above regarding Makeup. Often you want to use serializers like the authors of assert_value suggest, of course.

One could also generalize the concept to imething like property testing, but in which the property is visual inspection by a human (subsequent runs will compare to the reference result, of course).

EDIT: removed last paragraph because I’ve just reread the links and it turns out I had misinterpreted it.

As somebody who generally hates unit tests because of exactly the problem this solves…thank you

assert_value is asynchronous. While assert_value waits for user input the other tests continue to run. Only interactions with the user are synchronous.

You will want to split things across multiple tests. Each test should invoke Makeup on a single input file and assert the output. You will want to separate tests like this because test terminates when the first assert in it fails and to get maximum parallelism. That’s why my code above generates one test per input file instead of a big test for all files. I should have made this more clear.

The way we could do it is something like GIT_EXTERNAL_DIFF. That would make sense.

This should certainly be made more clear! I haven’t tried your library yet but I was under the completely wrong impression that it would pause waiting for input.

Hm… Doesn’t seem ideal for the use case of visually inspecting Makeup’s output. In that case I don’t care about diffs, I want to look at the rendered HTM. But in any case that’s probably not a good idea. Inspecting the list of tokens is probably much more useful than the rendered HTML and that can be done with serializers like your webpage example.

My idea of using a web browser was that you could compare things visually that might be hard to do textually. I admit I don’t have a concrete example for that (even Makeup isn’t probably the best example). Code that generates images is an obvious case, of course. You’d probably never do that in Elixir, so that’s not a problem, of couse. I guess I was fantasizing on an architecture that could be useful someday for something without thinking too hard about it.

I agree, the intent is quite similar. Thanks for the links!