SeanShubin

SeanShubin

Is this a better way to test Elixir?

This is a followup to my earlier attempt here

The difference I am experimenting with here is to see if swapping out entire modules rather than individual functions is less intrusive on the code under test.

The problem of separating determinism from nondeterminism has been solved for decades with dependency injection,
and until someone is able to point out a better way,
I am sticking with the theory of dependency injection for now.

The idea is to pass modules into the parameter list in production,
and make stubs and fakes look like modules for test.

This turned out to be much more difficult than I expected, but I am not sure if that is because it is not easy in Elixir, or if I am not aware of the proper language constructs.
I ended up dynamically naming my fake and stub modules to keep their state isolated from each other.
I refactored duplication with a macro.
I pasted the IOFake module to the end of test/test_helper.exs because I couldn’t figure out another way to get the other tests to see it.

Although I would prefer a solution that is more simple and less clever,
this does still seem, to me, to be an objectively superior way to test non-determinism in Elixir code than what I have seen so far in the ecosystem.
I can run my tests in parallel without worrying about colliding state.
I have taken complete control of all non-determinism.
The code under test does not need to change much (I could even leave the parameters capitalized, making them look like modules instead of parameters).
I don’t have to make excuses regarding why this or that is too hard to test, I just test everything.

I only did IO.puts here, but you can see how this can be extended to any non-determinism.

For those of you that are interested in this kind of thing, do you see any flaws in my logic?
Or perhaps I missed a better way because I don’t know elixir that well yet?

defmodule ConcurrentA do
  def send_line(line, io) do
    io.puts(line)
  end
end
defmodule ConcurrentB do
  def send_line(line, io) do
    io.puts(line)
  end
end
defmodule ConcurrentATest do
  use ExUnit.Case, async: true

  test "send line" do
    Fake.with_io fn io ->
      ConcurrentA.send_line("hello", io)
      ConcurrentA.send_line("world", io)
      actual = io.get_lines()
      expected = ["hello", "world"]
      assert expected == actual
    end
  end
end
defmodule ConcurrentBTest do
  use ExUnit.Case, async: true

  test "send line" do
    Fake.with_io fn io ->
      ConcurrentB.send_line("hello", io)
      ConcurrentB.send_line("world", io)
      actual = io.get_lines()
      expected = ["hello", "world"]
      assert expected == actual
    end
  end
end
# Pasted at the end of test_helper.exs
defmodule IOFake do
  defmacro __using__(_) do
    quote do
      def loop(lines) do
        receive do
          {:get_lines, caller} ->
            send(caller, Enum.reverse(lines))
            loop(lines)
          {:puts, line} ->
            new_lines = [line | lines]
            loop(new_lines)
          {:stop, caller} ->
            IO.puts("stopped #{__MODULE__}")
            send(caller, :stopped)
          x -> raise "unmatched pattern #{inspect x}"
        end
      end
      def puts(line) do
        send(__MODULE__, {:puts, line})
      end
      def get_lines() do
        send(__MODULE__, {:get_lines, self()})
        receive do x -> x end
      end
      def start() do
        process = spawn_link(fn -> __MODULE__.loop([]) end)
        Process.register(process, __MODULE__)
        IO.puts("started #{__MODULE__}")
      end
      def stop() do
        send(__MODULE__, {:stop, self()})
        receive do x -> x end
      end
    end
  end
end

defmodule Fake do
  def with_io(f) do
    id = System.unique_integer([:positive])

    {_, io, _, _} = defmodule String.to_atom("IOFake#{id}") do
      use IOFake
    end

    io.start()
    result = f.(io)
    io.stop()
    result
  end
end

First Post! Switch mode

JohnnyCurran

JohnnyCurran

If this is what you want to do, I think you should read:

Doing something like:

def my_io do
  Application.get_env(:io, :io_impl)
end

then in your app config you can:

# config/prod.exs
config :my_app, :io_impl, IO

# config/test.exs
config :my_app, :io_impl, MockIO

And then in your app code you can:

# some_business_logic.ex
my_io().puts("some text")

If you’re leaning towards passing around side effecting modules to all of your code - I would reconsider

Unless you are talking about testing the actual implementations of IO, File then I can’t be of more help. But if you want to make your tests have deterministic side effects then I think the article will be helpful for you

Most Liked

benwilson512

benwilson512

Author of Craft GraphQL APIs in Elixir with Absinthe

OK here we go!

https://github.com/benwilson512/mox_demo/blob/main/lib/mox_demo.ex

This has your main function. As you can see I have swapped out the System IO and File to the general IO behavior module I define here. mox_demo/lib/io.ex at main · benwilson512/mox_demo · GitHub

As I note in that module, it’s a bit simplistic to put all the IO functions in one module. In the real world you’d probably define multiple behaviors, but I wanted to keep this simple.

Then we have the test:

https://github.com/benwilson512/mox_demo/blob/main/test/mox_demo_test.exs

As you can see I can define test specific expectations. In the end I didn’t even really need to manage any state, since the ability to chain expectations meant that when you call eg monotonic_time twice I just define two expectations, one with the first value, and one with the second.

I don’t have additional tests in there right now but these expectations are 100% isolated from any other tests that would be happening.

One nice consequence of this approach is that backend being parameterized you still get strong compiler and language server support since it resolves to a compile time value.

This is barely scratching the surface of what’s possible though and ironically, a lot of what’s possible is made possible by how minimalist Mox is. Here is the same test set up a different way where it’s more interactive. Basically instead of setting up all the IO ahead of time and calling main(), we put main in a task and then have it basically “talk to” our test process as if it is the IO world:

https://github.com/benwilson512/mox_demo/blob/main/test/interactive_test.exs

All I’m doing here is combining core Elixir primitives like send and receive with Mox. Mox doesn’t really have any state to track here other than knowing which functions to call for this particular invocation of main. Then the “state” of the IO is in this case managed by my test process itself by sending messages to the mock at the appropriate times.

And of course you’ll note that you can just run mix test in the project root directory and these two test files will run at the same time and do not interfere. Both approaches are perfectly valid. I tend to favor the Mox.expect approach when I just have some outside system that I need a value or two from, or that I want to assert we pushed a value to. I tend to use the second approach if I’m writing more of a “simulator” test where there is some IO heavy piece of code and I want to step through it. This particular example makes that look sort of verbose but that setup block is generic. Once you write your little stub you’re done, and each test just gets to really focus on the interaction between the function under testing and the IO.

SeanShubin

SeanShubin

Thanks so much for putting the work in, that was really helpful for my prototyping. You can see my various attempts here:

Now that I got the mechanics down I can try to figure out which I like best and why, but I am too tired for analysis now. I wanted to post my results here in case anyone else is interested.

ityonemo

ityonemo

Mox lets you do concurrent testing. Each expect is bound to the lifetime of the test, and processes can figure out which test they map to to get where they need to go.

Instead of passing modules around, a better choice is to assign the module to a module attribute so it will be set at compiletime. This is extremely unobtrusive and basically replaces Modulename for @modulename. If you need to use a default impl aways except for when an expect is explicit, you can use stub_with to set it.

If you are wondering about the mechanics of how concurrency works with mox, here:

Last Post!

SeanShubin

SeanShubin

Thanks, I have made the update.

I have posted my analysis here:
https://forum.elixirforum.com/t/analysis-of-testing-styles-in-elixir/57329

I found the intention of my analysis sufficiently different than my intention for the discussion here to warrant a separate thread.

Where Next?

Trending in Discussions Top

AstonJ
As the title says, please share what you’ve been up to with Elixir. Whether that’s been learning it, looking into it, making stuff with i...
2976 91332 914
New
byu
@chrismccord : I just saw the Extract AGENTS.md from Phoenix.new into phx.new generator commit to the phoenix project. My initial shotgu...
New
arcanemachine
I was working on an Ecto migration and I needed a timestamp. So, for the nth time, I looked up the different data types for timestamps, a...
New
AstonJ
Just a general thread to post chat/news/info relating to AI/ML stuff that may be relevant for Nx now or in the future. Got anything to sh...
New
type1fool
I just stumbled on a newly redesigned elixir-lang.org. :tada: It looks like @Software_Mansion did the work, and I think it is generally a...
New
juhalehtonen
There has been a thread to discuss the Stack Overflow Developer Survey on this forum every year since 2018, so here’s yet another one for...
New
alexslade
Fly’s CEO posted this recently - Turn And Face The Strange · The Fly Blog It says that Fly is going all-in on sprites, which is a worry ...
New

Other Trending Topics Top

JesseHerrick
Hey, I’m Jesse and I’m the main contributor behind Dexter, a full-featured, lightning-fast Elixir LSP optimized for large codebases. It s...
New
jimsynz
Beam Bots (or just BB for short) is a framework for building fault-tolerant robotics applications in Elixir using familiar OTP patterns. ...
New
Damirados
Hello everyone. After busy few months I am happy to announce v0.1.0 of Emerge & Solve. They are GUI (Emerge) and State management (S...
New
ausimian
Emily is an Elixir library that runs Nx computations on Apple’s MLX. Install it as the default Nx backend and Nx, defn, Axon, Nx.Serving,...
New
akoutmos
@hugobarauna and I (Alex Koutmos) have been hard at work on writing a book on Nerves that takes you from simply blinking LEDs to building...
New
zachdaniel
Introducing AshStorage! Attachment and file management that slots directly into your resources :smiling_face_with_sunglasses: I had hope...
New

We're in Beta

About us Mission Statement