I wanted to write down some of the main guides I use when writing Elixir tests, so I have summarised them in blogpost.
Furthermore, I think that mocking must be destroyed.
Destroyed and replaced with cassettes, recorded from actual calls. Iād love to see a generic recorder for āanyā elixir module/function call and response to etf or similar format. This has only become more important as use of llms grows.
1 Like
I really donāt like describe "foo/2", test that the requirements are met, not the implementations. Iād rather group the tests by goals though I still use the function name for helpers or convenience public API functions.
Also fully on board for simpler database seeding, one of our staff engineers introduced ex_machina everywhere but to me it is just a headache factory.
1 Like
@subjectmodule attribute for module under test
I picked this up from you a few years ago when looking at your projectionist config and still use it to this day!
I think it depends on the ālevelā youāre testing at. I largely agree with you and yet I still find myself doing this all the time. As a minor note: the one thing I donāt do is include the arity. In my experience, it always gets out of sync and people donāt seem to notice.
Re: ex_machina, I agree it makes little sense in Elixir. With FactoryBot your factories would run through validation (which you touched on) and callbacks and Iāve often found myself essentially re-implementing the context function in my factories.
Ashās generators have hit the right note for me since you get a changeset_for_* function for every domain (context) function. Ash has callbacks which you do need to call manually from generators, but this often a good thing if the callback isnāt essential for your tests (no silver bullet). Iāve always felt keeping as much logic as possible in changesets is good practice in Phoenix anyway so changesets could be used in plain olā Phoenix test as well.
2 Likes
If you prefer context function with MyApp.Blog.create_comment_for instead of ExMachina, and dislike mocking, how do you deal with complex context functions that needs to do a multitude of side effects such as:
- capture audit log changes via carbonite
- validate the user permission
- trigger webhook or API calls
- etc ā¦
I understand wanting to test via the application functions but doesnāt it make the test suite much more complex and heavier?
Of course, it does not mean that you even need ExMachine, you could directly insert via Repo but you wrote that you prefer context functions so very curious if/how you handle that.
I tend to agree with everything short of:
In Elixir we have bunch of the mocking libraries out there, but most of them have quite substantial issue for me - these prevent me from using
async: truefor my tests.
Huh? The argument sounds to me as āwe have several orange suppliers, the most of them supply apples instead, thatās why I donāt buy oranges at all.ā To my best knowledge, each project needs one mocking library, and mox is carefully designed to allow running everything in parallel with a help of nimble_ownership.
In finitomata there is a whole testing framework allowing to test all the possible transitioning scenarios in FSM (async, in parallel, of course), and without mox it could not be even possible, because finitomata allows ādeterminedā transitions, which are executed instantly once the āfromā state for them is reached.
2 Likes
As I said in the article - these are guides not rules. There are situations when you need to break rules, but I find that often using any mocking library (whether it is Mox, Repatch, Patch, or any other) while sometimes allows doing tests in async manner, it often quickly break when you want to cross process boundaries. And if you want to use such mocks there, then you need to resolve to manually passing function anyway, which is not really different from āfunctional DIā.
However, as these are guides (again), there obviously will be points where you will break them. It is normal and expected. These are meant to be āhigh level suggestionsā which I prefer to use unless I am forced to do otherwise.
How to test effects or the integrations like with external services and such?
Repatch mentioned 
Repatch works with cross-process scenarios, but you need to find the processes to add to async group, which is the biggest problem. If you could point out the difficult parts, maybe I could add some functionality
That is the problem - you need to find processes to add them to group. And if you can do so, then IMHO patching isnāt a problem, but at the same time, in such cases you can often find a solution without mocking anything at all.
I would argue that in my personal experience the sole fact that difficulties do arise is a clear sign of the bad design, namely a superfluous coupling. Mocking here plays the same role as Boundary lib for static code.
I agree, but perhaps I could introduce helpers to add all processes linked to some process to group, or maybe add all children of some supervisor, or maybe introduce something based on tracing. I am curious to hear your pain points
I do not remember any particular issues right now. Repatch is my go-to library when I am forced to do mocking so a lot of kudos for this project. However, whenever I can, I simply try to avoid the need, and I am pretty successful with that so far.
2 Likes
Agree with describe with function name but I also include the specific input data case (with foo and bar) because I prefer to test those in isolation to avoid cascading failures and ballooning setup.
I have tried strictly using contexts for setup and just find it too cumbersome, although I grant it seems more pure and I also avoid libraries for struct generation that ācompeteā with contexts. I write whatever code is most convenient for the case. Sometimes thatās a context function, mostly itās a private helper calling insert!. Have yet to be bitten by this.
I am on team ādonāt write application code for the sake of testsā so I just mock stuff. Itās a tradeoff, but Iād rather my code running in prod as as univocal and expressive as possible, even if that costs some cycles. But thatās from writing code for 25 years before AI came along, I find some of my personal āguidelinesā bending quite a bit recently.
3 Likes
Man, donāt get me started on mocking. Iāve been looking at poor designs for years and the only way to test anything at all is by using mocks. You talk about DI in relation to mocking, which is great, but I also miss programming against a contract which can be achieved through protocols.
4 Likes
I personally donāt hate mocking, but I limit myself exclusively to Mox, because itās painful enough to use. It takes some boilerplate and manual setup and ownership tracking, and this is exactly the kind of incentive I need to not overuse it.
People program in Java for exactly the same reason
2 Likes
nah, people write Java because they know it and it pays their bills
2 Likes
I would ask the opposite question. Do you want a test suite that checks your code in states the system can actually reach in production, paying for it with some slower setup, or do you want fast tests that run on data no business flow could actually produce? āJust write a valid setup by handā is easy to say. So is ājust write bug free codeā. If either was realistic, we would not need tests in the first place 
Context functions carry a contract, and for anything non trivial that contract is much simpler than putting rows into tables by hand. Letās say you write a chat and you want a thread with a first message. The function is something like start_thread(sender, recipient, initial_message). Internally it creates the thread, links both users, inserts the message, marks it read for sender and unread for recipient. One line in your test setup vs. a lot of ceremony to recreate that exact state directly in the DB.
So is it really āmore complexā? Iād say writing it is less complex. There is more code running, yes. But that code is exactly what reduces the space of valid fixture states down to what the system can actually produce. To me that is the whole point, not a cost.
Heavy, yes, in absolute terms. But ex_machina is not free either, and I think people underestimate this. :org has one :user and has many :project, and :project has :created_by and :approved_by assocs, both pointing at :user. Declare those naively as associations and a plain insert(:project) will create three users where you wanted one. To prevent that you start writing factory logic that walks through the other associations to reuse an existing user, and the simple user: build(:user) one liner turns into something messy inside the factory file. The complexity does not disappear.
My personal take: have a centralized test setup layer built on functions, and by default let it call context functions. When a specific context function is too heavy and is used in most tests (for example, users are created as :pending and must be activated through a separate flow, which is correct for the business but annoying as a prerequisite for every test), I add a lighter setup helper for that case, with a short comment explaining why it skips the normal flow.
3 Likes
How do you test interactions with external services without mocks? How is DI different from mocks in Elixir? I am using mocks and patching for side effect testing and isolation of tests to make them async and I am curious about your experience
Depends what external services we are talking about. In many cases instead of mocking I prefer writing stub. So for HTTP services I will run local HTTP service that I can point to, for DNS service I will implement simple resolver that runs locally. And so on.
1 Like
