Blog Post: Some Elixir Testing Tricks

Learning Resources Blogs & Podcasts
, , ,
1

Testing is an important part of any modern piece of software. But writing tests can quickly become an exercise in frustration, with tons of repeated code, duplicate setup routines, and cumbersome assertions. ExUnit, Elixir’s testing framework, is ultimately just pure Elixir, and so we can extend it using more elixir features. Isolated setup units that can be mixed and matched, configuration via tags, and custom assertions are trivial to add to a test suite, and can save tons of developer headache

24 Likes
2

Thanks, pretty useful. I already used part of these so I am definitely stealing using the others.

2 Likes
3

Very nice article!

2 Likes
4

Very nice article with some brilliant tips. Thanks for writing and sharing it.

Gonna bookmark it so next time I have to write a test suite (soon!) I can practice some.

2 Likes
5

Very well written and useful. Thx for sharing!

It prompted me to review the documentation of the ExUnit.Case module. I’ve never used tags before, but the examples you provide in the blogpost are very clear. I’d even argue that you could easily improve the existing docs on tags with an example from your post. I wouldn’t have guessed that the tags are passed into the setup callbacks, and take advantage of them (combined with the pattern matching superpowers) like you showed. Super handy indeed. It could as well have worked the other way: setup callbacks are run first, and the tag value is merged into the context last. That’s not clear from the current documentation on tags, afaik.

While reading up on the tag docs I also noticed that you can do:

@tag :admin

which is equivalent to:

@tag admin: true

This spares another four characters :slight_smile:

The last part on how to generate tests with the for-comprehension prompted another question: can you run the tests in a way that it prints out all the test names that are run, also if they pass? When you’re generating tests like that, it’s nice to see the generated test name at least once, to see if it will make sens if they fail. I know you should start with a failing test, but still… :slight_smile: Would be nice to have a formatter that prints more than just a green dot for every passing test. There is a CLI option to override the default formatter, but I didn’t find any bundled formatter that I could pass in.

3 Likes
6

I’ll look into improving them in an Elixir PR this weekend!

Indeed. But since tags are handled by the compiler (they’re just attributes, you can register your own for anything via Module.register_attribute/3. This is how some tools like Absinthe register their documentation and such.

Interestingly, ExUnit also provides a register_attribute/2, which can be used to register attributes for use only in tests. They show up under context.registered, i.e.:

@fixture :foo
# registers as
context.registered.fixture == :foo

There’s also a sibling one for describe attributes, register_describe_attribute/2 and one for modules, register_module_attribute/2, which can be used as described.

I’ve used the TAP formatter in the past to check test names. You can also follow the red-green-red pattern of testing, where you make the tests deliberately fail on the first run, to check the setup. If the test is empty you’ll get a warning about it not being implemented, and if you just want to force a fail you can assert false.

4 Likes
7

I very much echo the “great article” sentiments and thank you for making me aware of TAP!

I have a somewhat similar helper as your assert_html for LiveView tests to narrow down text within a certain element which I find myself doing a fair bit often:

defmacro assert_within(lv, selector, text_or_regex) do
  quote do
    assert unquote(lv)
           |> element(unquote(selector))
           |> render() =~ unquote(text_or_regex)
  end
end

assert_within(lv, ".comments", "Product was definitely broken when it arrived, please send another.")
3 Likes
8

I’ve written and used similar macros before. One thing that I’ve seen some do, and have mixed feelings about, is directly grabbing the lv variable from the calling site. This is nice because it lets you write really compact tests


click("Edit Profile")

assert_view("h1", "Editing profile")

But the downside is that they’re “magical,” they grab the LV or HTML out of their call site with no apparent way of doing so to the caller. Depending on your teams proficiency and feeling towards this, it can be a major downside.

I’m gonna collect some more of these tips into a second article :smile:

1 Like
9

Yep—it crossed my mind to do that and considered mentioning it here but didn’t want to give anyone any ideas :sweat_smile: I think it’s a bit gnarly and of course it stays pipeable if you directly accept the dependency. I’ve abandon every idea I’ve had like this after shooting myself in the foot with Ruby over the years. You technically don’t even need a macro to make it work as it is, but it’s a little bit simpler sorta.

3 Likes
10

We use some custom click/form fill macros that pull it out of the current space at my current job, but my team is a two man team, and we both agreed that it was better than having to write lv all over the place.

All our assertions still require you to pass a html by hand.

1 Like
11

Hey if a team can decide and stick to it that’s totally cool! I’m solo right now but don’t trust myself not to die by a thousand cuts :sweat_smile: I also want less to document and explain if anyone else ever joins the project so I’ve been avoiding even the most seemingly innocuous things.

1 Like
12

It was mostly to reduce the large maintenance burden of writing

view |> element("button", "Create post") |> render_click()

everywhere (over 10k tests!)

We defined two versions, however, one that does “magic” and one that takes the view in as its first argument, so you can override as you need.

Looking at the calls, we use the explicit one in a couple places, and the implicit one everywhere else.

And its noted in the documentation using an Admonition/Nutrition facts thing, and failure to have a view variable set causes a compile error.

1 Like
13

Ya I get it. I sometimes feel my tests get a little verbose with lv or view piping into everything so it wouldn’t take much to get me to buy into this if I were on a team that wanted to.! I was really interested in this project. I still am but I got really distracted by work and it just kind of slipped my mind.

14

Yeah that one is pretty cute; works by passing around a tuple of the things you’d need for each one. Pretty clean way of doing it without much macro abuse

15

Speaking of macros, I’m curious why you use macros for your helpers instead of regular functions?

16

Thats mostly out of a bit of laziness, to allow the module that’s defining the macros to not have to import the parts of ExUnit that does the assertions.

If you wanted to do them as pure functions, you’d have to import ExUnit.Assertions into the module you’re defining them in.

I’d written up an explanation on what I thought was the right answer, and even went to update the blog post to show you could use functions, but thought I should test it, and then on the first run of the test suite, got errors from no function named assert. So there you have it!

2 Likes
17

Thanks. I asked because I’ve been writing this kind of helpers for at least a couple of years now and I never thought about writing them as macros, so I was wondering if there was a deeper reason behind your choice.

2 Likes
18

This is my current command to run the tests:

mix test --failed --max-failures 1 --seed $(odo -c _build/seed.txt) && \
mix test --stale  --max-failures 1 --seed $(odo -c _build/seed.txt) && \
odo _build/seed.txt

odo is a simple counter that reads or increments a number from a file.

So what is does is:

  • as long as I have a failing test, it will only run that test, thanks to the seed
  • when this test passes, it will now fail repeatedly on the next failing test
  • when all my previously failing tests pass, it will run all the stale tests. If one of them fails, the loop will start over.
  • if everything passes, we increment the seed to preserve a minimum of entropy like designed in ExUnit.

A simple shortcut for vscode:

  {
    "key": "ctrl+alt+u",
    "command": "workbench.action.terminal.sendSequence",
    "args": {
      "text": "mix test --failed --max-failures 1 --seed $(odo -c _build/seed.txt) && mix test --stale --max-failures 1 --seed $(odo -c _build/seed.txt) && odo _build/seed.txt\n"
    }
  },
2 Likes