Automating Tests in Elixir Projects

What happen to using behaviours and implement a test module.

I also find that all these libraries makes it way more complex then it should be and forget an important lesson to teach around architecture and simplicity.

I think the Elixir documentation deserves a simple dependency free architecture guide vs a million libraries in search of a problem.

Another point would also be, if you want to mock a HTTP response, why would you not setup a complete web-server for it, so you have a much control as possible, TestServer and Bypass comes to mind, where TestServer really shines with it’s support for WebSockets and flexibility in request patterns that let’s you modal highly complex cases with ease.

using behaviours and implement a test module

Is that what I called Manual with Behaviours? Where you define a behaviour, and provide a real and a test implementation?

I like that too - it’s explicit and easy to reason about, with reasonable tradeoffs. Although it still raises the question of how you swap in the test module? I think the application environment is an oft-used mechanism, but I don’t find it especially simple or elegant… Curious if you’ve got another approach?

I share the instinct to keep things simple and avoid unnecessary dependencies. It’s a solid baseline. At the same time, I find that carefully chosen, narrow libraries are a big time-saver - they do one thing well and are easy to read and reason about. It’s always a tradeoff though.

On HTTP mocks vs real servers: I tend to see external test harnesses as more fitting for E2E tests. I haven’t covered that space yet in the blog, but definitely plan to. The focus so far is on unit tests, since they form the foundation of a testing strategy. That’s not to say they’re the only part. A test server like you describe sounds like a great fit when I get to that layer.

Testing is one of those areas with strong opinions and multiple valid approaches. My blog is more of a “random walk” than a “thorough search” through testing techniques - just things I’ve seen and tried in the wild. Always happy to hear how others are doing it so I can keep learning, if you feel like sharing?

It’s simple to reason about it, as it is global state. As for elegant, IMO simple things almost always lead to elegant solutions, and I think injection by config is elegant enough for simple cases.

I used to mock http requests too, however after using once bypass, it’s crystal clear that this is a much better approach. Not only you can check in detail http requests with exactly zero modifications to your source code, but you can also go into advanced topics such as https. Doing something like that with mocks will require quite a substantial amount of time investment to get right, not to mention that mocking tends to be on the dangerous side if you are not doing it right.

I think that such kind of libraries (ecto sandbox is another great example) are amazing and this is the right way to approach testing. Obviously this cannot be universally applied as such implementations are involved, however they amount to the best testing experience and results.

One problem with using the application environment for injection is that tests don’t run concurrently, right? I shouldn’t argue it’s not simple though - you’re right, it’s a global key-value store. But I find it surprising when runtime code reads from it not for user-configurable settings, but purely as a mechanism for injection. It feels like an awkward fit to me (as elucidated in the blog).

Personally, I find it easier to reason about when a process gets its dependencies from its own process hierarchy — and as a bonus, it enables async tests. But: Just opinions. It’s a wide tent, and many ways to success. What matters most must be that the team is effective and having fun within whatever is the chosen setup.

I’m smiling at the HTTP discussion - it’s revealing more nuance than I’d considered: I also don’t want to mock HTTP, but I’ve typically placed those kinds of tests higher up, e.g. E2E. But Bypass sounds great - I’ve added it to my to-investigate list so I can make sure I understand it properly.

Ultimately I’m open to test doubling internal modules when they grow too large or too entangled to fit in our heads, because I’ve seen devs really appreciate the clarity that comes from such decoupling. So I’ve got a net-positive view on this kind of approach - as long as we use the simplest (not always easiest) ways to do it. And I appreciate the engagement here, it’s a nuanced topic.

If you plan on using separate modules, yeah, but generally you have a 2 module setup: the one that calls the external api and the one that does mocks.

Give it a try for sure. This is no new concept and it’s much more easier to reason about compared to the complex setup that mocks require.

Mocking internal logic for me is a hard no, this is one of the big reasons I try to avoid mocks whenever I can. The point of tests is to ensure correctness of your system at different levels. If you are mocking actual business logic, then you are introducing tests that can result in false positives which may cause more problems than they solve, as the code is a dynamic media that is always changing.

but generally you have a 2 module setup

I appreciate your point here, and to me what you describe is complex. It’s cognitive load. A delayed gotcha of sorts. I don’t say that to convince you of course, but to illustrate the spectrum of the testing topic.

On internal mocks, it never fails to elicit a strong response - and no disrespect at all, I actually find it fascinating how wide our “testing tent” is. What I describe and what you’re advocating (perhaps loosely: mockist vs. classicist) both have passionate, experienced camps behind them.

I’ve seen systems where a few well-placed mocks helped a team move forward, just as I’m sure you’ve seen systems suffer from mock abuse. I like to think there are reasonable compromises in the middle - but that only works if we can talk openly about how to mock well, so we can better decide when it makes sense to do it.

I sometimes wonder if each camp is just wired differently - like trying to work left-handed when you’re right-handed - and I suspect the best solution is typically Conway/architectural: put us on two different teams

Anyway, I know even that framing won’t sit comfortably with everyone, and I respect that. It’s part of what makes testing such a rewarding and sometimes difficult topic to explore.

I am not entirely sure what you mean by camp, I’ve never read even a single book on testing and I certainly don’t have any time to do that now . I am entirely practice based on this topic.

I’ve learned to write tests by mistakes mostly. I’ve started with projects that I didn’t write tests for when I was a beginner, then by writing brittle tests, abusing mocks etc. These days I am always adapting tests to the current codebase and team I work with, but things such as mocking business logic need to be motivated very thoroughly for them to be used, as I’ve previously said, modifications to code that are not expected can lead to false positives and a distorted reality between testing environment and code that runs in production.

Mocking internal code is a huge red flag, it tells everyone that there is fundamental architectural problem, or lack of experience.

In the end, programming is all about data transformations. To put it simple: input → output

So when we test, then we want to assert that we got the expected output from the input.

With that in mind we can design our system in such a way that we only care about changing the input, that input can come from a lot of places.

I saw you mention unit testing and E2E. People tend to conflate all these terms. 99% of the time programmers write integration test, E2E test is a subset of integration test. All your Ecto test are integration test. If a function has side effects then the only way to test it is by an integration test.

Unit test requires a function without side effects.

Regardless of this pedantickery lets just agree that tests should be simple to reason about and if you feel that mocking is the right answer, then you properly have a bigger problem at hand.

I think quite the contrary: test doubles are invaluable in testing, including internally. They help isolate systems and keep tests focused. I used the word “mock” earlier, but didn’t mean to imply a specific kind of test double — any chance that caused some misunderstanding?

Internal test doubles let us test specific behavior without pulling in all underlying dependencies. Dismissing them overlooks how they can simplify complex systems and prevent growing dependency chains. Not every project needs them — but when used well, they help many devs better understand system boundaries.

This isn’t to my knowledge a controversial position in the wider testing world. Mockist vs. classicist doesn’t capture all the nuance we’re touching on here, but it hints at the broad and diverse approaches under the test automation umbrella. For better or worse, we’ve got to learn to see eye to eye

I think we’re talking past each other, again terms are being conflated, if not made more complex.

When I’m talking about mocking internal code, then what typically happens is that the mock changes your code. If you need to modify your business logic in a perverse goal of test isolation to the extreme. Then you fundamentally have bigger problem at hand, which is basically experience. It takes a long time for any new developer to learn how to cut through the pedantic BS, especially when it comes to testing which is some of the most opinionated discussions you can have.

Regardless my point being a more fundamental software engineering 101. Everything we do is data transformations. input → output.

We don’t need to make it more complex. When we understand this, then we can build beautiful simple systems, which can grow into complex systems that are easier to understand.