How do I test things like IO, File, and System?

New to elixir, looking for guidance regarding how to test things like IO, File, and System

Whenever I learn a new language, I always make sure I know how to test it by implementing the following “Hello, world!” application.

defmodule HelloApp do
  def main() do
    time_unit = :microsecond
    microseconds_before = System.monotonic_time time_unit
    target = System.argv |> List.first |> File.read!
    IO.puts "Hello, #{target}!"
    microseconds_after = System.monotonic_time time_unit
    microseconds_duration = microseconds_after - microseconds_before
    IO.puts "Took #{microseconds_duration} microseconds"
  end
end

I intentionally choose the requirements for this application because they have the most extreme combination of easy-to-write yet hard-to-test that I can think of.
As it is in any language, the trick is to isolate the side effects behind an abstraction of some kind.
With Elixir this seems harder to do than normal, but perhaps that is simply because I don’t know the proper language constructs to hide the side effects behind an abstraction.
I don’t see a way to swap out System, IO, and File with stub versions because they are sitting in a global namespace.
I did look at ExUnit.CaptureIO, but rejected it for 2 reasons.
First reason is that as the capture is happening globally, I become vulnerable to one test affecting another.
Second reason is that CaptureIO is not generalizable to File and System.

Ideally I would like each test to validate the implementation in a sandbox independent of other tests with all the side-effecting modules swapped out.

I was able to get 100% test coverage by inverting the dependencies, so it looks like this may indeed be the right answer.
However, since this is my first ever Elixir program, I want to check with people with more Elixir experience than me to make sure I am on the right track.

What do you think of this solution to getting IO, File, and System under 100% test coverage?
Can you guide me to a better solution?

For the implementation, I invert each side-effecting dependency

defmodule HelloAppInverted1 do
  def main(collaborators) do
    system = collaborators.system
    file = collaborators.file
    io = collaborators.io
    time_unit = :microsecond
    microseconds_before = system.monotonic_time.(time_unit)
    target = system.argv.() |> List.first |> file.read!.()
    io.puts.("Hello, #{target}!")
    microseconds_after = system.monotonic_time.(time_unit)
    microseconds_duration = microseconds_after - microseconds_before
    io.puts.("Took #{microseconds_duration} microseconds")
  end
end

For production, I configure to use the real thing:

defmodule HelloAppEntry1 do
  def main() do
    system = %{
      :monotonic_time => &System.monotonic_time/1,
      :argv => &System.argv/0
    }
    file = %{
      :read! => &File.read!/1
    }
    io = %{
      :puts => &IO.puts/1
    }
    collaborators = %{
      :system => system,
      :file => file,
      :io => io
    }
    HelloAppInverted1.main(collaborators)
  end
end

For testing, I configure to use stubs:

defmodule HelloAppInverted1Test do
  use ExUnit.Case
  @moduletag timeout: 1_000

  test "say hello to world" do
    tester = create_tester(
      %{
        :command_line_arguments => ["configuration.txt"],
        :remaining_monotonic_time_values => [1000, 1234],
        :file_contents_by_name => %{
          "configuration.txt" => "world"
        },
        :lines_emitted => []
      }
    )

    tester.run.()

    assert tester.lines_emitted.() == ["Hello, world!", "Took 234 microseconds"]
  end

  def create_tester(initial_state) do
    state_process = create_process(initial_state)

    monotonic_time = fn time_unit ->
      send(state_process, {:get_monotonic_time, time_unit, self()})
      receive do
        x -> x
      end
    end

    argv = fn ->
      send(state_process, {:get_argv, self()})
      receive do
        x -> x
      end
    end

    read! = fn file_name ->
      send(state_process, {:get_file_contents, file_name, self()})
      receive do
        x -> x
      end
    end

    puts = fn output_string ->
      send(state_process, {:puts, output_string})
    end

    lines_emitted = fn ->
      send(state_process, {:get_lines_emitted, self()})
      receive do
        x -> x
      end
    end

    system = %{
      :monotonic_time => monotonic_time,
      :argv => argv
    }
    file = %{
      :read! => read!
    }
    io = %{
      :puts => puts
    }
    collaborators = %{
      :system => system,
      :file => file,
      :io => io
    }

    run = fn ->
      HelloAppInverted1.main(collaborators)
    end

    %{
      :run => run,
      :lines_emitted => lines_emitted
    }
  end

  def create_process(state) do
    spawn_link(fn -> loop(state) end)
  end

  def consume_monotonic_time(state) do
    [time_value | remaining_time_values ] = state.remaining_monotonic_time_values
    new_state = Map.replace(state, :remaining_monotonic_time_values, remaining_time_values)
    {new_state, time_value}
  end

  def append_line(state, line) do
    new_lines_emitted = [line | state.lines_emitted]
    Map.replace(state, :lines_emitted, new_lines_emitted)
  end

  def loop(state)do
    %{
      :command_line_arguments => ["configuration.txt"],
      :remaining_monotonic_time_values => [1000, 1234],
      :file_contents_by_name => %{
        "configuration.txt" => "world"
      },
      :lines_emitted => []
    }
    receive do
      {:get_lines_emitted, caller} ->
        send(caller, Enum.reverse(state.lines_emitted))
        loop(state)
      {:get_monotonic_time, :microsecond, caller} ->
        {new_state, monotonic_time_value} = consume_monotonic_time(state)
        send(caller, monotonic_time_value)
        loop(new_state)
      {:get_argv, caller} ->
        send(caller, state.command_line_arguments)
        loop(state)
      {:get_file_contents, file_name, caller} ->
        file_contents = state.file_contents_by_name[file_name]
        send(caller, file_contents)
        loop(state)
      {:puts, line} ->
        new_state = append_line(state, line)
        loop(new_state)
      x ->
        raise "unmatched pattern #{inspect x}"
    end
  end
end
  • edit, I said production twice, the last snippet was for testing, not production
1 Like

Hey @SeanShubin welcome!

While true, I think there’s more to say here. As a functional language, the first thing to do wouldn’t be to reach for an abstraction, it would be to reach for extraction (I couldn’t resist). Basically, work to extract the code with side effects from the logic of your program.

Then much of your logic can be tested just with pure inputs and outputs. In particular the values you’re reaching for like the path to read or write a file to should definitely not be produced in the same function that actually does the writing or reading to the file.

4 Likes

So is the idea that you just try to keep the computations and conditions away from the side-effects, and accept that the side effects won’t be under test coverage?

Not necessarily no, you can still test those, and tools like Mox — Mox v1.1.0 are popular for setting up abstractions around side effects when you want to test end to end behavior. I just wanted to emphasize that the more you can push complex logic into the side effect free part of the code the easier it is to both reason about (due to immutable data) and also test.

Given that the majority of Elixir users are writing web apps, it’s also worth noting that Ecto does some particularly fancy stuff to make tests easier when it comes to database access. Basically it uses transactions to create little sandboxes for each test so that you can do the actual database writes but still not impact other tests.

So in this particular example, if I extract all the side effects, I get this:

Code without side effects

defmodule HelloApp2 do
  def main(target, microseconds_before, microseconds_after) do
    microseconds_duration = microseconds_after - microseconds_before
    [
      "Hello, #{target}!",
      "Took #{microseconds_duration} microseconds"
    ]
  end
end

Code with side effects

defmodule HelloAppEntry2 do
  def main() do
    time_unit = :microsecond
    microseconds_before = System.monotonic_time time_unit
    target = System.argv |> List.first |> File.read!
    microseconds_after = System.monotonic_time time_unit
    lines = HelloApp2.main(target, microseconds_before, microseconds_after)
    Enum.each(lines, &IO.puts/1)
  end
end

Which forces me to rely a lot less on unit testing and a lot more on other kinds of tests than I am used to.
However, to your point about the majority of Elixir users writing web apps, this may not matter so much.
I am glad you brought up Ecto, as I know I am going to be needing to figure out how to test database interactions in a phoenix framework application soon.
It is good to be pointed to a possible solution for when it comes time to address that particular need.

FWIW, what you are describing is more in the lane of integration / end-to-end testing than unit testing. Having pure functions that don’t rely on side effects and testing them with various inputs is what I would do in an imperative / OOP language as well, not only in an FP one (like Elixir). Testing those and/or their modules is what’s unit testing.

If you anticipate wildcard events – like time jumps due to DST (which just happened last night in Europe) – then craft a specific test that mimics the scenario?

As for files, I can partially side with you because it’s not a one-minute job to imitate conditions with no disk space remaining, exhausted file descriptors etc. but with the help of the already mentioned Mox – or Patch – you can find the underlying Erlang code or dig in the docs for the right error responses when these things happen and just return them straight away in a mock.

(Me and many others in the Elixir community use this technique to a crushing success, very often, e.g. when you have an API client contacting 3rd party servers you can just directly return 404, 500 or a validation error in your mock. Or 429 for too many responses etc. You can exercise the unhappy paths very easily. Not super quickly, mind you, it’s still a job where you must apply rigor and cover your expected error conditions exhaustively. But it’s possible and it’s done regularly in commercial projects.)

I don’t think there’s a magic silver bullet here. As programming matures as a profession / discipline, people at large mostly understand that 99% of your code should be pure and the rest 1% should be well-covered with integration tests utilizing mocking.

2 Likes

Do we put testing certain kinds of behavior in the lane of integration testing because that is where it belongs? Or are we just so used to testing those kinds of behavior with integration testing that we have not taken the time to think about how we could have designed the code in such a way that it is easier to test? If we are actually doing “functional” programming, then how is it that we have gotten in the situation that we need so many integration tests?

The way I am testing in my initial example is actually functional. Not in a subjective “feels more functional to me” way, but in an objectively demonstrable way. By arranging the code the way I did, I am describing a relationship between inputs (independent variables) and outputs (dependent variables) such that every possible input maps to exactly one output. I am doing this explicitly by passing all of the inputs in the “collaborators” variable. What I am not doing, is taking a set of inputs that can be defined by the caller, and also interacting with a different set of inputs that are pulled by name from a global namespace, such that they are hardcoded and can not be defined by the caller, therefore abandoning my ability to control my inputs and easily test any scenario I need to write code for. I do not believe there is such a thing as code that is hard to test, only code that is badly designed, which can always and everywhere be fixed by some form of inverting a dependency and hiding the parts you can’t control behind some form of abstraction.

I do understand there are practical reasons not to code in a way so different that you can’t get support from existing tooling and community, so I don’t want to be misunderstood as combative or saying anyone is doing anything incorrectly. My intent is to encourage people to think more precisely about what they mean by “functional” and especially about what are the real reasons why we think we need an integration test instead of a unit test. My starting point is 100% test coverage for everything with 100% unit tests, 0 integration tests, and 0 end to end tests. I deviate from this as need arises, but not lightly and not unless I can precisely articulate, categorize, and narrowly scope the reasons for the slower and less reliable tests. I am still brand new to Elixir, that sample I posted is my first ever Elixir program, so while I don’t want to be so arrogant that I know better than those more experienced than me, I don’t want to accept ideas blindly either, I want to make sure I understand the reasoning behind those ideas. Also as someone brand new, I am better equipped to detect hidden assumptions that more experienced people might be so used to that they are blind to them.

I did have a look at the beam file format, and my impression is that it should be possible to stub out all side effecting calls by only modifying the atoms table (AtU8) and import table (ImpT). This would allow stubbing out all side effects, which means you could even test your side-effecting code with unit tests, and only need to rely on integration testing when you actually want to test the thing you are integrating with rather than testing your own code. Does anyone know of existing Elixir tooling that already does this?

Are you sure? How specifically is the style of inverting dependencies to be passed in as collaborators not solving the problem of getting the Elixir code under 100% test coverage with unit testing alone. I am not saying it is magic, but if there is a way in which this style is not solving the problem I want to know specifically what that is.

You’re right, this has been a solved problem for literally decades. Dependency injection is the very obvious answer here. Unfortunately, most functional languages tend to be pretty bad at it.

It doesn’t help that people often don’t realize that there even is a problem (I’ve experienced the same attitude in the Ruby community as well). And those who do end up making various mocking libraries, each with a different set of disadvantages. As much as the FP community likes to complain about design patterns in OOP that result from shortcomings of the language, this is a great example of the same problem happening in FP as well.

On the other hand, the fact that FP languages tend to be bad at DI has resulted in some very interesting research like algebraic effects, which can be used for many other things as well.

The thing is, for all practical purposes, I already have this. For stubbing out side effects involving interaction with APIs Mox basically is what you’re asking for with DI, and for interacting with the DB you can either do the same thing or you can use Ecto sandboxes. Personally I prefer Ecto sandboxes because unless you’re doing very simple things with a DB the stubs would be pretty complicated to code.

GitHub - jjh42/mock: Mocking library for Elixir language uses meck (an erlang library) to do actual code substitution which seems to be what you’re referring to in your last paragraph.

Languages with a strong reliance on DI aren’t my main wheelhouse so I’m sure I could be missing some nuance, but to me Mox seems like it is exactly what you’re looking for.

2 Likes

Thanks! I will definitely check it out.

1 Like

As a guy why was on both sides of the fence, I can’t differentiate between dependency injection in Java and mocking in Elixir.

Yes, the way it’s done is mechanically different but the achieved result is the same.


I’m not certain what exactly is @SeanShubin trying to encourage us to think about, having in mind that both imperative / OOP and FP languages have very good tools to tackle the problem of side effects and misbehaving 3rd party libraries. Even though I’ve carefully read his long comment above, my only impression is that he wants to change the meaning of words that are already well-established. :person_shrugging:

And yep, Mox or Patch will do the job. Mock too, I’ve used that successfully as well.

It is not my intention at all to change the meaning of well established words. I am encouraging people to think about the meaning of the words they use, and use them with intention and precision. The meaning of the word “function” is very well established with a precise mathematical definition. The solution I demonstrated was a functional approach according to this already well established definition created a long time ago by many people who had a lot more collective time to think about it than I ever could.

What I see is a lot of is people thinking they are doing “functional” programming, while at the same time getting inputs from a global namespace rather than getting those inputs from the parameters to their function, or thinking I am not doing functional programming because I have some locally scoped mutable state or an imperative style, while definitionally it is still a function because the same inputs always get the same outputs with no observable side effects.

I can’t control your impressions of what I am saying, I can only attempt to be as clear as I possibly can be. I just want to make sure observers of this thread are aware that the particular impression of me wanting to redefine well established terms is a construct of your own mind that has nothing to do with what I was trying to say. I never said or implied that.

Sure. We can’t have 100% pure code without side effects however.

Maybe my mind is not on the same frequency as yours – I am just not sure what you are after in practical terms.

My aspirational goal is to get 100% of my Elixir code tested with unit tests alone. Code not run by the Erlang runtime system, such as SQL database queries, while certainly side effecting, is not Elixir code so I am fine with using integration tests for those as long as all of the Elixir code used to transmit the SQL to the database is unit tested.

In this sense, I believe my sample using dependency inversion did achieve 100% pure code with no side effects for every single bit of my Elixir logic, in a way that can be generalized to any kind of side effects. The only class that did not have test coverage was the class that configured which production implementations to use, which is fine by me because that configuration contains no application behavior, just direct pass throughs to the underlying implementations. It is so simple it it could be generated. All of the application behavior is easy to stub out for unit tests, no need for integration tests for Elixir code and no need to skip test coverage on anything.

That said, I am also willing to be pragmatic about my aspirational goal, and am willing to pull back on it in cases where there is good reason and a decent alternative. I don’t want to be writing Elixir code in a style completely alien to the rest of the Elixir community just for the sake of principle.

I am also willing to explore alternative suggestions that can get me close to my aspirational goal through means I might not be able to think of on my own due to my inexperience with Elixir. This is why I am grateful for the suggestions to look into Mox, Ecto, and jjh42/mock. I might have missed these on my own. Once I have had a chance to review all of the solutions, I feel like I can make an informed decision regarding what tradeoffs between functional purity and pragmatism are sensible. I don’t want to presume I have to make a tradeoff, I want to make sure any tradeoffs I make are for well understood and good reasons.

Hello @SeanShubin and welcome.

Your target audience is a bit unclear here. Your writing style is attracting responses from seasoned Elixir developers and seasoned Elixir developers are well aware they are not working in a purely functional language. Erlang was developed to solve a practical problem and its functional properties emerged as a consequence of that. It’s never been academic. Elixir developers generally favour pure functions for the most part and push all side-effects to a particular boundary which isn’t necessarily the top boundary (depending on how you think about your system).

I’d say go for it if you want. You will generally find people around here encouraging others to code however they see fit. If you look through some prominent open source Elixir projects there are all sorts of different styles.

3 Likes

I think that is a consequence of me being new to Elixir while at the same time having such ambitious goals regarding testability. When I start coding in a programming language that is new to me, I want to get certain foundations nailed down first.

I have found the ability to write code in a pure functional style has very little to do with the language. I can write any program in a language such as Java in 100% pure functional style and Java is not even trying to be a functional programming language. I have also seen code in Haskell, a pure functional programming language, where the side-effecting IO Monad has so permeated the entire code base that I can’t even comprehend what it means to call that code functional.

The reason why I am so focused on precision regarding what it means to be functional, is that I am trying to parse out the justification for when to take a functional vs non-functional approach. Sometimes it is because of language limitations that ruin the aesthetics of certain styles, sometimes it is the way people are used to coding, sometimes there are well established conventions, and I guess that comment was a challenge to see if the only reason the functional approach was not chosen in a particular case is because it doesn’t occur to people that it is even possible. If you tell me I can’t have pure functional, or maybe it doesn’t make sense to strive for pure functional, that may very well be true, I just want to know why.

I am not seeking any more clarification right now though, I got enough suggestions to keep me busy. I am glad you let me know about the history of Elixir being practical and becoming functional as a side effect, not the other way around.