SeanShubin
New to elixir, looking for guidance regarding how to test things like IO, File, and System
Whenever I learn a new language, I always make sure I know how to test it by implementing the following “Hello, world!” application.
defmodule HelloApp do
def main() do
time_unit = :microsecond
microseconds_before = System.monotonic_time time_unit
target = System.argv |> List.first |> File.read!
IO.puts "Hello, #{target}!"
microseconds_after = System.monotonic_time time_unit
microseconds_duration = microseconds_after - microseconds_before
IO.puts "Took #{microseconds_duration} microseconds"
end
end
I intentionally choose the requirements for this application because they have the most extreme combination of easy-to-write yet hard-to-test that I can think of.
As it is in any language, the trick is to isolate the side effects behind an abstraction of some kind.
With Elixir this seems harder to do than normal, but perhaps that is simply because I don’t know the proper language constructs to hide the side effects behind an abstraction.
I don’t see a way to swap out System, IO, and File with stub versions because they are sitting in a global namespace.
I did look at ExUnit.CaptureIO, but rejected it for 2 reasons.
First reason is that as the capture is happening globally, I become vulnerable to one test affecting another.
Second reason is that CaptureIO is not generalizable to File and System.
Ideally I would like each test to validate the implementation in a sandbox independent of other tests with all the side-effecting modules swapped out.
I was able to get 100% test coverage by inverting the dependencies, so it looks like this may indeed be the right answer.
However, since this is my first ever Elixir program, I want to check with people with more Elixir experience than me to make sure I am on the right track.
What do you think of this solution to getting IO, File, and System under 100% test coverage?
Can you guide me to a better solution?
For the implementation, I invert each side-effecting dependency
defmodule HelloAppInverted1 do
def main(collaborators) do
system = collaborators.system
file = collaborators.file
io = collaborators.io
time_unit = :microsecond
microseconds_before = system.monotonic_time.(time_unit)
target = system.argv.() |> List.first |> file.read!.()
io.puts.("Hello, #{target}!")
microseconds_after = system.monotonic_time.(time_unit)
microseconds_duration = microseconds_after - microseconds_before
io.puts.("Took #{microseconds_duration} microseconds")
end
end
For production, I configure to use the real thing:
defmodule HelloAppEntry1 do
def main() do
system = %{
:monotonic_time => &System.monotonic_time/1,
:argv => &System.argv/0
}
file = %{
:read! => &File.read!/1
}
io = %{
:puts => &IO.puts/1
}
collaborators = %{
:system => system,
:file => file,
:io => io
}
HelloAppInverted1.main(collaborators)
end
end
For testing, I configure to use stubs:
defmodule HelloAppInverted1Test do
use ExUnit.Case
@moduletag timeout: 1_000
test "say hello to world" do
tester = create_tester(
%{
:command_line_arguments => ["configuration.txt"],
:remaining_monotonic_time_values => [1000, 1234],
:file_contents_by_name => %{
"configuration.txt" => "world"
},
:lines_emitted => []
}
)
tester.run.()
assert tester.lines_emitted.() == ["Hello, world!", "Took 234 microseconds"]
end
def create_tester(initial_state) do
state_process = create_process(initial_state)
monotonic_time = fn time_unit ->
send(state_process, {:get_monotonic_time, time_unit, self()})
receive do
x -> x
end
end
argv = fn ->
send(state_process, {:get_argv, self()})
receive do
x -> x
end
end
read! = fn file_name ->
send(state_process, {:get_file_contents, file_name, self()})
receive do
x -> x
end
end
puts = fn output_string ->
send(state_process, {:puts, output_string})
end
lines_emitted = fn ->
send(state_process, {:get_lines_emitted, self()})
receive do
x -> x
end
end
system = %{
:monotonic_time => monotonic_time,
:argv => argv
}
file = %{
:read! => read!
}
io = %{
:puts => puts
}
collaborators = %{
:system => system,
:file => file,
:io => io
}
run = fn ->
HelloAppInverted1.main(collaborators)
end
%{
:run => run,
:lines_emitted => lines_emitted
}
end
def create_process(state) do
spawn_link(fn -> loop(state) end)
end
def consume_monotonic_time(state) do
[time_value | remaining_time_values ] = state.remaining_monotonic_time_values
new_state = Map.replace(state, :remaining_monotonic_time_values, remaining_time_values)
{new_state, time_value}
end
def append_line(state, line) do
new_lines_emitted = [line | state.lines_emitted]
Map.replace(state, :lines_emitted, new_lines_emitted)
end
def loop(state)do
%{
:command_line_arguments => ["configuration.txt"],
:remaining_monotonic_time_values => [1000, 1234],
:file_contents_by_name => %{
"configuration.txt" => "world"
},
:lines_emitted => []
}
receive do
{:get_lines_emitted, caller} ->
send(caller, Enum.reverse(state.lines_emitted))
loop(state)
{:get_monotonic_time, :microsecond, caller} ->
{new_state, monotonic_time_value} = consume_monotonic_time(state)
send(caller, monotonic_time_value)
loop(new_state)
{:get_argv, caller} ->
send(caller, state.command_line_arguments)
loop(state)
{:get_file_contents, file_name, caller} ->
file_contents = state.file_contents_by_name[file_name]
send(caller, file_contents)
loop(state)
{:puts, line} ->
new_state = append_line(state, line)
loop(new_state)
x ->
raise "unmatched pattern #{inspect x}"
end
end
end
- edit, I said production twice, the last snippet was for testing, not production
Where Next?
Trending in Questions
Other Trending Topics
Latest on Elixir Forum
Sponsor Spotlight
Phoenix, LiveView, multimedia pipelines, distributed systems. 400 engineers, Elixir since 2017. Available for hire.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #library
- #deployment
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #podcasts
- #javascript
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #ai
- #elixirconf-us
- #blog-post
- #elixir-ls
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #hex
- #security
- #metaprogramming
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Elixir, Erlang & BEAM conferences from Code Sync - join us in 2026!
Practical resources that improve the lives of professional developers.
Producing high quality Elixir screencasts since 2017.
**Building the future of software with AI, trust, and human connection.**
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
Showing Posts 1 to 10- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
benwilson512
Hey @SeanShubin welcome!
While true, I think there’s more to say here. As a functional language, the first thing to do wouldn’t be to reach for an abstraction, it would be to reach for extraction (I couldn’t resist). Basically, work to extract the code with side effects from the logic of your program.
Then much of your logic can be tested just with pure inputs and outputs. In particular the values you’re reaching for like the path to read or write a file to should definitely not be produced in the same function that actually does the writing or reading to the file.
SeanShubin
So is the idea that you just try to keep the computations and conditions away from the side-effects, and accept that the side effects won’t be under test coverage?
benwilson512
Not necessarily no, you can still test those, and tools like Mox — Mox v1.2.0 are popular for setting up abstractions around side effects when you want to test end to end behavior. I just wanted to emphasize that the more you can push complex logic into the side effect free part of the code the easier it is to both reason about (due to immutable data) and also test.
Given that the majority of Elixir users are writing web apps, it’s also worth noting that Ecto does some particularly fancy stuff to make tests easier when it comes to database access. Basically it uses transactions to create little sandboxes for each test so that you can do the actual database writes but still not impact other tests.
SeanShubin
So in this particular example, if I extract all the side effects, I get this:
Code without side effects
Code with side effects
Which forces me to rely a lot less on unit testing and a lot more on other kinds of tests than I am used to.
However, to your point about the majority of Elixir users writing web apps, this may not matter so much.
I am glad you brought up Ecto, as I know I am going to be needing to figure out how to test database interactions in a phoenix framework application soon.
It is good to be pointed to a possible solution for when it comes time to address that particular need.
dimitarvp
FWIW, what you are describing is more in the lane of integration / end-to-end testing than unit testing. Having pure functions that don’t rely on side effects and testing them with various inputs is what I would do in an imperative / OOP language as well, not only in an FP one (like Elixir). Testing those and/or their modules is what’s unit testing.
If you anticipate wildcard events – like time jumps due to DST (which just happened last night in Europe) – then craft a specific test that mimics the scenario?
As for files, I can partially side with you because it’s not a one-minute job to imitate conditions with no disk space remaining, exhausted file descriptors etc. but with the help of the already mentioned Mox – or Patch – you can find the underlying Erlang code or dig in the docs for the right error responses when these things happen and just return them straight away in a mock.
(Me and many others in the Elixir community use this technique to a crushing success, very often, e.g. when you have an API client contacting 3rd party servers you can just directly return 404, 500 or a validation error in your mock. Or 429 for too many responses etc. You can exercise the unhappy paths very easily. Not super quickly, mind you, it’s still a job where you must apply rigor and cover your expected error conditions exhaustively. But it’s possible and it’s done regularly in commercial projects.)
I don’t think there’s a magic silver bullet here. As programming matures as a profession / discipline, people at large mostly understand that 99% of your code should be pure and the rest 1% should be well-covered with integration tests utilizing mocking.
SeanShubin
Do we put testing certain kinds of behavior in the lane of integration testing because that is where it belongs? Or are we just so used to testing those kinds of behavior with integration testing that we have not taken the time to think about how we could have designed the code in such a way that it is easier to test? If we are actually doing “functional” programming, then how is it that we have gotten in the situation that we need so many integration tests?
The way I am testing in my initial example is actually functional. Not in a subjective “feels more functional to me” way, but in an objectively demonstrable way. By arranging the code the way I did, I am describing a relationship between inputs (independent variables) and outputs (dependent variables) such that every possible input maps to exactly one output. I am doing this explicitly by passing all of the inputs in the “collaborators” variable. What I am not doing, is taking a set of inputs that can be defined by the caller, and also interacting with a different set of inputs that are pulled by name from a global namespace, such that they are hardcoded and can not be defined by the caller, therefore abandoning my ability to control my inputs and easily test any scenario I need to write code for. I do not believe there is such a thing as code that is hard to test, only code that is badly designed, which can always and everywhere be fixed by some form of inverting a dependency and hiding the parts you can’t control behind some form of abstraction.
I do understand there are practical reasons not to code in a way so different that you can’t get support from existing tooling and community, so I don’t want to be misunderstood as combative or saying anyone is doing anything incorrectly. My intent is to encourage people to think more precisely about what they mean by “functional” and especially about what are the real reasons why we think we need an integration test instead of a unit test. My starting point is 100% test coverage for everything with 100% unit tests, 0 integration tests, and 0 end to end tests. I deviate from this as need arises, but not lightly and not unless I can precisely articulate, categorize, and narrowly scope the reasons for the slower and less reliable tests. I am still brand new to Elixir, that sample I posted is my first ever Elixir program, so while I don’t want to be so arrogant that I know better than those more experienced than me, I don’t want to accept ideas blindly either, I want to make sure I understand the reasoning behind those ideas. Also as someone brand new, I am better equipped to detect hidden assumptions that more experienced people might be so used to that they are blind to them.
I did have a look at the beam file format, and my impression is that it should be possible to stub out all side effecting calls by only modifying the atoms table (AtU8) and import table (ImpT). This would allow stubbing out all side effects, which means you could even test your side-effecting code with unit tests, and only need to rely on integration testing when you actually want to test the thing you are integrating with rather than testing your own code. Does anyone know of existing Elixir tooling that already does this?
SeanShubin
Are you sure? How specifically is the style of inverting dependencies to be passed in as collaborators not solving the problem of getting the Elixir code under 100% test coverage with unit testing alone. I am not saying it is magic, but if there is a way in which this style is not solving the problem I want to know specifically what that is.
Sorc96
You’re right, this has been a solved problem for literally decades. Dependency injection is the very obvious answer here. Unfortunately, most functional languages tend to be pretty bad at it.
It doesn’t help that people often don’t realize that there even is a problem (I’ve experienced the same attitude in the Ruby community as well). And those who do end up making various mocking libraries, each with a different set of disadvantages. As much as the FP community likes to complain about design patterns in OOP that result from shortcomings of the language, this is a great example of the same problem happening in FP as well.
On the other hand, the fact that FP languages tend to be bad at DI has resulted in some very interesting research like algebraic effects, which can be used for many other things as well.
benwilson512
The thing is, for all practical purposes, I already have this. For stubbing out side effects involving interaction with APIs Mox basically is what you’re asking for with DI, and for interacting with the DB you can either do the same thing or you can use Ecto sandboxes. Personally I prefer Ecto sandboxes because unless you’re doing very simple things with a DB the stubs would be pretty complicated to code.
GitHub - jjh42/mock: Mocking library for Elixir language · GitHub uses
meck(an erlang library) to do actual code substitution which seems to be what you’re referring to in your last paragraph.Languages with a strong reliance on DI aren’t my main wheelhouse so I’m sure I could be missing some nuance, but to me Mox seems like it is exactly what you’re looking for.
SeanShubin
Thanks! I will definitely check it out.