Please take my post with a pinch of salt, as I’ve read 2 books on Elixir, but have no experience of it in production/OSS otherwise.

After painfully going through similar questions you’re raising, I’ve arrived at the following so far:

1. Elixir processes are not OOP classes, so we can’t expect to test them the same way (e.g. create a structure of several objects, perform actions and observe side effects).
2. Pure/functional Elixir code is easy to test.
3. Mocking/stubbing actors is hard. Seen this both in Elixir and in Orleans on .NET.

Therefore, I’ve found it’s most practical to:

1. Have as much logic as possible in the pure functional modules. Cover these with extensive unit tests.
2. For process-level testing, start the application (like mix test does by default), and send test messages/observe side effects as the application is running. Most likely it means having a DB running as part of your build job etc.

#2 assumes that the processes/services are very slim and delegate all decisions to the pure code. That’s not always easy as messages that get passed around are usually intertwined with the business logic.

I’m not sure I’ve made my peace with the above approach yet, but this is what the toolset has been pushing me forward. Larger projects on Github that I’ve looked at (Phoenix, Nostrum) all seem to be following similar philosophy.

I think we’re in agreement. It’s best if you can cover every module separately and then all the separate test combined test the entire system.

With respect to running the database, that is not ideal but not that bad either. However, the challenge with that is when you have an interaction that goes across multiple processes; how do you know the DB driver has performed the insert you want to test, without a sleep?

In my experience, particularly if you’re doing an “integration test” you would not mock the database at all. Ecto sandboxes are designed to work with multiple processes. Start your processes you need for your test, put Ecto in either the shared mode or use allowances, and do a real “end to end” test.

I have a few principles I try to follow when testing processes / sets of processes:

• Try to start processes in the test
  • isolated from anything else
  • no singletons or dependency on global state
• Application started processes are global state (best not to be depended on, unless they provide isolation on their own (e.g. ecto))
• Do not assert on processes internal state. Don’t look into the black box.
• Assert on things observable to the outside world using public APIs of the processes or interactions with other public resources
• Strongly prefer waiting on being messaged over sleeps.
• Don’t forget that the test is an isolated process, which can receive messages.
• If you want to test implemenation details used within a process, extract to a function and unit test the function.
• Consider stateful property tests. They’re involved, but also powerful. Can the the right tool depending on the context.
• Rearchitecting to aid testing is not a bad idea.

Imo this is a fallacy. The usecase for behaviours is to provide some level of interface where multiple implementations are to be used. If you have one implementation running, that’s hard to test, and a different one meant to aid in testing that’s multiple implementations. Even if an implementation happens to only be used in testing it’s just as bad if the implementation starts to drift apart with what is expected from the implementation as it would be for one used in production.

I think the general arguments about unit tests vs integration tests apply here as well.

Yes, and add sleeps to wait for the insert, which is undesirable. I don’t want to mock the database, I want to know when it has performed its tasks.

This looks like an actual fallacy namely circular reasoning:

1. You need a behavior because without one, it’s hard to test stuff
2. When it’s hard to test stuff, you need a behavior

Example

• There is a process A that generates telemetry as a side effect
• There is a module that subscribes to this telemetry and casts it to process B
• Process B holds some state that can lead to events being filtered. This is strongly tied what events generated by A look like
• Process B saves the events to a database

Of course I can test separately whether

• A generates the correct telemetry
• Telemetry leads to a cast
• Make the functions in B public and test whether they result in the correct database records

That also means

• Copying the telemetry content across tests, because we need to test whether A sends it correctly, and then use it as input for B
• Making a factory for the above
• Or more likely, certain events will not be tested fully

While your other points are well taken I don’t see a solution for an end-to-end test of the above that doesn’t involve sleeps.

Since when? Repo.insert is synchronous, when the function call is completed the record is in the DB.

No it doesn’t, Elixir behaviours are basically a form of programming by contract. They are not something that will ruin your life, they are used to (a) increase clarity on what does a certain agent in your program do and (b) help with mocking if you are so inclined.

That mention baffles me too, the only case where I would see uncontrolled concurrency happening is when somebody would use something like GenServer.cast/2 and this points to a bad design, as that function doesn’t guarantee that the message was received by the process.

Let’s try to maintain the level of the discussion. genserver cast is a normal OTP function which serves many purposes and is used very often.

Obviously if you are calling a function which calls insert directly or performs a cast which leads to calling it. But once there is an asynchronous step somewhere, whether insert itself is synchronous doesn’t matter

I think we should take a step back here because your comments strike me as a bit academic. Maybe we should discuss your particular hurdles when trying to test something concrete?

Discussions like these rarely are fruitful because everything carries tradeoffs – there’s no one single perfect solution. Though it’s also true there are a number of solutions that are objectively worse than others. If that’s your goal here – to uncover a subset of better solutions then OK but f.ex. arguing that “Repo.insert can actually be asynchronous” is not productive and won’t lead us to anywhere enlightening.