Good Elixir TDD resources?

[Fl4m3Ph03n1x]

Andrea Leopardi and I are currently writing a book on testing elixir for PragProg. While it does not emphasize TDD, I’m a strict test-driver. I’m not sure what resources need to be dedicated specifically to TDD if you are already familiar with the concepts from another language.

Let me know how I can help and I’m happy to. I do not know of any specific guides or posts.

The thing I miss the most would be spies and stubs ( looking at sinon.js ). Most people end up creating mocks when in reality they don’t need them. It just happens they don’t know about spies for example.
I am a believer that when it comes to designing tests, code and architecture, the simplest solution is usually the best. That’s why in over 2 years of programming I have never used a mock.
In Elixir I find that the solution for everything is just a mock ( mocks should be used carefully ).

Another thing I miss are the basic concepts that made ( at least for me) TDD viable, such Dependency injection and Inversion of control. I know I how to them JavaScript style, but as I said before, I want to write Elixir instead of writing JavaScript code using Elixir.

A section detailing any of these would clearly up my interest as I have found little information regarding the application of such techniques in the Elixir community.

I have also recently learned about Elixir’s love for property testing which I am new at ( coming from other backgrounds it is the first time I hear about it). A comparison between TDD and PT or how they can be combined together would be interesting to me.

Do note these are my personal opinions based on my personal struggles with Elixir. I have been a seasoned developer in other languages, but coming here I feel there is a lot of adaption needed.

I may not be a good representative of your target audience, you may not be writing a book for people who moved to Elixir after years of JavaScript or something else.

---

Already searched the forum with tdd as keyword? You will find BDD / TDD criticized . About the ruby community and testing: see links to DHH’s criticism in that thread also (I at least think I pasted them there).

Everyone is entitled to their opinion and as I stated, TDD / BDD is not the holy grail. It may not save you from your demons, but I assure you it has saved me from many ugly demons over the years. It works for me and for the work I do, so I’ll keep defending

[peerreynders]

I know I how to them JavaScript style , but as I said before, I want to write Elixir instead of writing JavaScript code using Elixir .

At the risk of stating the obvious:

• When dealing with function values nothing much changes. Pass in your fake function instead of the real function and have the fake do what it needs to do.

• However Elixir doesn’t have Objects. In JavaScript Objects conflate (mutable) state and behaviour. In Elixir state is stored in immutable data structures while behaviour resides in modules.

So it’s:

someValue = state.method(...params); // spread params array over arguments

versus

new_state = Module.function(old_state, params) # params is a parameter key list

So in Elixir the “fake object” situation may require a fake data structure (though at times you can simply use the real one) and will require a fake module, which means that your mockable boundary cannot hard code the module so that we can write:

new_state = apply(Module, :function, [old_state, params])

with the rough JavaScript equivalent:

someValue = (state.method).apply(state, params);

Now in Elixir that Module value has to come from somewhere and typically that is resolved at compile time (something that JavaScript doesn’t have) by grabbing the Module from the compilation environment - i.e. compilation in preparation for running tests versus running the application.

This is essentially what Mocking and Explicit Contracts and Mox are based on.

Consequently choosing mockable boundaries in Elixir is a much more deliberate process (hence “Explicit Contracts”) than it typically is in JavaScript with the ad hoc creation of injectable spies (where state has customized behaviour already bound to it).

(Another example)

[cdegroot]

so I’ll keep defending

I wonder whether building contractors have to defend using a double mitre saw over a skilsaw for certain jobs; Similarly, when I worked in the pharmaceutical industry, I never had lab managers argue the virtues of liquid over gas chromatography because they have their clear uses. I know that these sorts of discussions happen everywhere but us programmers seem particularly attached to just a subset of all possible tools.

TDD, BDD, test-after-the-fact, the-code-is-so-simple-I’m-not-even-gonna-test-it - all tools in your toolbox. Use them carefully; the trick is knowing what to apply when ;-). Usually, during my rare full days of coding, I go through them all.

[StefanHoutzager]

With TDD you can write and rewrite a function until it passes your test while you are not understanding what you are doing. It is important to understand what you need. When you can formulate (above code level) what you need you can write maximally comprehensible code (and design the best tests).

Programmers who advocate writing tests before writing code often believe those tests can serve as a specification. Writing tests does force us to think, and anything that gets us to think before coding is helpful. However, writing tests in code does not get us thinking above the code level. We can write a specification as a list of high-level descriptions of tests the program should pass—essentially a list of properties the program should satisfy. But that is usually not a good way to write a specification, because it is very difficult to deduce from it what the program should or should not do in every situation.

Testing a program can be an effective way to catch coding errors. It is not a good way to find design errors or errors in the algorithm implemented by the program. Such errors are best caught by thinking at a higher level of abstraction. Catching them by testing is a matter of luck. Tests are unlikely to catch errors that occur only occasionally—which is typical of design errors in concurrent systems. Such errors can be caught only by proof, which is usually too difficult, or by exhaustive testing.

[Egis]

In Elixir I find that the solution for everything is just a mock ( mocks should be used carefully ).

Can you give a specific example on what exactly are you mocking?

[Fl4m3Ph03n1x]

Databases, APIs, Cache, basically entire services. The idea here (a according to the elixir community ) is to create a contract and then create an implementation for that contract as well as a mock ( or a fake implementation ).

Then depending on the environment you either use the real object or a mock that implements the contract. ( do note that by definition, your real object also implements the contract ).

I am still not convinced with mocks, I pretty much prefer to simply pass the functions I am going to need down the line ( Functional DI, shout out to @peerreynders for the video link ).

The issue with this is that your higher level functions will have a ton of dependencies, no matter what … So I guess this is where the argument for mocks could take place …

[Egis]

Databases, APIs, Cache, basically entire services.

Why do you need test/mock database (or 3rd party api, cache and etc.)? Given that you’re not writing database, 3rd party api or cache library.

You either receive correct response from them or you don’t. You need to mock data, not services.

But that’s just my (incorrect?) opinion and one of the main reasons I’ve fell in love with fp/elixir.
It’s just functions (grouped in modules) that work on data - simple and elegant.

[Fl4m3Ph03n1x]

You need to mock them so you can do what we call “contract tests”. In short you need to:

1. make sure you call the contract with the correct parameters
2. make sure you handle the data returned by those who implement the contract correctly

If you don’t understand what i mean by any of this, I recommend you watch “Integration tests are a scam”.

[Egis]

1st item is not possible in elixir, since you can’t call function with wrong parameters (it won’t compile). By “wrong” I mean too much/too little params.

This leaves us to 2nd item - making sure you handle data correctly. That’s why you mock data, not services.

[StefanHoutzager]

Btw:
I googled “integration tests are a scam” because that title seemed strange. The title appeared a mistake: https://blog.thecodewhisperer.com/permalink/clearing-up-the-integrated-tests-scam

[Fl4m3Ph03n1x]

1st item is not possible in elixir, since you can’t call function with wrong parameters (it won’t compile). By “wrong” I mean too much/too little params.

I am not talking about a signature ( you even have spec to help with that ) I am talking about calling the contract with params that make no sense. Like wrong values and such.

Really, go watch the video conference. It will help you understand the context of this discussion:

• https://www.youtube.com/watch?v=VDfX44fZoMc

[Egis]

I am talking about calling the contract with params that make no sense. Like wrong values and such.

Property testing to the rescue!

Edit: also dialyzer.

[Fl4m3Ph03n1x]

@Egis Dialyzer won’t save you. In fact, if you are using Elixir 1.7.X it will even hurt you, as it doesn’t even work.
Even if ti did work, it wouldn’t matter. Dialyzer can’t understand that a method being called with “foo” should be instead called with “baz”. For Dialyzer, both are strings and the function’s signature is correct, so it passes.

As for property testing, I am not convinced. I am actually reading a book that mentions it ( The Little Elixir and OTP Guidebook ) but the examples are rather contrived and perfected to make sense using property testing. The applications to the real world are … considerably more limited.

Still, this is a discussion for another topic. I don’t think property testing is a good as people say ( feel free to create a new discussion referring me if you want to try and convince me) but I am still studying it’s potential applications.

This discussion is about TDD and good resources for those who value it.

[NobbZ]

In fact, if you are using Elixir 1.7.X it will even hurt you, as it doesn’t even work.

Please do not tell it like this, as dialyzer works with 1.7.x if the OTP versions match. This is not an issue of dialyzer or elixir, this is an issue of installing the correct things in the correct versions.

For Dialyzer, both are strings and the function’s signature is correct, so it passes.

This sounds to me as if the types haven’t specified correctly.

As for property testing, I am not convinced.

Sad. You should give it a go. For algorithms that have well defined properties this is a nice thing.

Eg. list == list |> reverse |> reverse or a + b == b + a or len(a++b) == len(a) + len(b). Having those is really nice. Of course it requires a different set of thinking than example based testing.

[Fl4m3Ph03n1x]

Please do not tell it like this, as dialyzer works with 1.7.x if the OTP versions match.

No it doesn’t: https://stackoverflow.com/questions/53427886/which-version-of-elixir-otp-erlang-can-we-use-with-dialyzer

This sounds to me as if the types haven’t specified correctly.

Dialyzer checks types. Doesn’t check values. I could have a function add return 1+1=3 and it would pass. Why? both are numbers, return a number and the function takes the correct number of parameters. It is still wrong. Dialyzer doesn’t know any better because it doesn’t check values. That’s the job of TDD. To say that Dialyzer can replace TDD is just flat out wrong. It can help, sure, but never replace.

Sad. You should give it a go. For algorithms that have well defined properties this is a nice thing.

Yo are correct. However I found two main issues:

1. Few times we have code that deals with nice algorithms that deal with very well defined properties. Sure, if you want to re-invent the wheel and reverse lists it will work fine but real life apps are messy and full of side effects and edge cases that don’t quite fit into PT.
2. The code you write with PT is actually … quite complex. I am not even talking about recursive generators nor trees, but even the most simple things, like applying the reverse algorithm, require you to code both the solution and the counter solution. If I don’t know my solution works well, how am I supposed to know my counter-solution used to test my solution does? It makes no sense to me ( PS: counter solution = inverse function, inverse algorithm, equivalent of the undo operation ).

I am not giving up on PT, not yet. I want to delve deeper into it, however at this point in time, I am still not convinced

But power to you if you make it work for you. Perhaps you have some resources I could use?

[OvermindDL1]

I am not even talking about recursive generators nor trees, but even the most simple things, like applying the reverse algorithm, require you to code both the solution and the counter solution.

The counter should generally be part of the API as well for simple property tests. However property tests are not just about reverse testing but about invariant testing, like have it generate inputs into a state system, test that the state of it is valid each time and make sure it ends at a proper place given the inputs, even without knowing the values you catch a significant amount of bugs.

But power to you if you make it work for you. Perhaps you have some resources I could use?

As a good example of state testing see the Elevator example for the C side of the QuickCheck library (which uses Erlang to do the testing) for how to think about it.

[peerreynders]

Aspect: Testing as One Driver of Design

For algorithms that have well defined properties this is a nice thing.

My exposure to property testing is minimal (e.g. worked through an elaborate a QuickCheck demonstration in Haskell) and for the time being Elixir Forum Property-Based Testing with PropEr, Erlang, and Elixir is a bit of a specialist topic for me to fully commit - but the reason it is staying on my radar is that I suspect that beyond the primary benefits it may impose beneficial design pressures on the code.

you find a rule that dictates the behavior that should always be the same no matter what sample input you give to your program, and encode that into some executable code—a property.

1. In order to use property based testing the STU has to be designed against somewhat coherent (and explicitly stated) rules that the tests can be based on (now if the same was only true for “business” logic). So if you want to leverage property based testing it seems the need for “rules” improves coherence. The flip side is that if you can’t be bothered to discover/refine those “rules”, you are not going to see the value in property based testing. With example based testing there is never any real incentive to unify fragmented, incoherent logic.

2. While this rant states that “You Don’t Need Referential Transparency” I’d be inclined to believe that property based testing would drive you towards preferring referential transparency.

However I still see property based testing as a rather specialized tool in the “testing” part of software development, likely useful for micro tests, possibly useful with collaboration tests, likely not appropriate for contract tests.

Some approaches to testing can be inefficient/ineffective (similar to the situation in process and documentation). Hypothetically property based testing should be more efficient than example based testing because it can cover more values with less effort. But the issue is that it can’t replace example based testing.

Given that developers will be exposed to example based testing first (and testing is a means to an end, rather than an end in itself - unless you are specializing in testing) they may never get exposed to property based testing.

Then there is the possibility that property based testing is only applicable and effective for code bases that already meet certain quality standards.

the examples are rather contrived and perfected to make sense using property testing

Which begs the question: could some software benefit from being “perfected” (i.e. designed) to be more amenable to property based testing and would that improve the software’s design from functional and maintenance perspective?

[csaintc]

Chiming in, as I am also a big fan of TDD but not fanatical in my implementation of it.

I have experienced the same thing @Fl4m3Ph03n1x , testing in Elixir is requires a lot of arcane knowledge that will not arise from getting into the code and writing the tests. Elixir lacks a popular opinionated testing framework and it is easy to do wrong, or to architect your code base to make testing a slog.

I’m working on a blog post about this exact subject, so I will share some of my notes.

Macros make the dream work

Keep in mind: Macros make the dream work! They let you write amazing abstractions for your code and I highly recommend getting VERY familiar.

There is a book by THE Chris McCord https://pragprog.com/book/cmelixir/metaprogramming-elixir

The (new) Facade Pattern

First and foremost, there is a pattern we have yet to name but is mentioned over and over in the community to enable clean mocking and testing. You can find resources on it here:

• https://www.youtube.com/watch?v=Ue--hvFzr0o&list=PLqj39LCvnOWaxI87jVkxSdtjG8tlhl7U6&index=8
• http://blog.plataformatec.com.br/2015/10/mocks-and-explicit-contracts/
• Use Mox for mocking: https://github.com/plataformatec/mox

Mocking using a fake server process

a good reference impl of this is bamboo.

• https://github.com/thoughtbot/bamboo/blob/master/lib/bamboo/test.ex
• https://github.com/thoughtbot/bamboo/blob/master/test/lib/bamboo/adapters/mandrill_adapter_test.exs

Other resources

Additionally, I do property based checking and macro away as much boilerplate as I can.

• read this book: https://pragprog.com/book/fhproper/property-based-testing-with-proper-erlang-and-elixir and use propcheck for a lot of my testing.
• There are a lot of good libraries here, look through them and see if any of them speak to you. Don’t be shy about forking and taking over maintenance of a project that has died and looks promising! I like your opinions and I think writing an opinionated framework would be great! https://github.com/h4cc/awesome-elixir#testing
• https://github.com/thoughtbot/ex_machina for generating test data
• Meck is controversial, but you can just decide to use it anyway: https://github.com/eproxus/meck

[joebew42]

Hi Pedro,

Probably I am bit late here, but I was just googling for “TDD Elixir” and I found this thread. I am deeply interested in topics like these.

Few years ago I started a Twitch Channel with the intent to show others how it is possible to apply practices like Clean Code, TDD and Refactoring as a support to learn a new programming language.

I decided to learn Elixir: starting from the basics of the language, the I did some programming session on Code Katas, and at the end I had the chance to push some little contributions to different open source projects, meet and learn from more experienced people. In general, an 100% nice experience! Here you can access to several examples of my journey in learning Elixir with TDD.

As a result, I wanted to create a talk (unfortunately it’s only available in Italian) where I shared the process that I followed to learn a new programming language doing TDD:

Eventually, I would like to propose an updated version of this talk, in english!

I remember also a talk from @gpad on TDD and Elixir, from the last Code Beam STO 2019

Now, after more than one year I started to work for a company where we play daily with Elixir and I learned something more about how is it possible to apply TDD (new tools, new libraries, what works, what don’t, etc …). And it’s time to share something, again.

I will give a speech at the next Beam Languages United - Stockholm Meetup with a practical example of TDD Outside-In in Elixir.

The session should be recorded and then uploaded on YouTube, from the organizers of the meetup. I will ask.

By the way, I will then upload an offline session on YouTube also, with only the desktop, the code and my voice. Something similar to this.

I am still learning, it’s an interesting journey and I am absolute available to share experience with anyone else interested in these topics.

[Fl4m3Ph03n1x]

Hello @joebew42 I am happy to know you are also interested in the Elixir meetup, I truly hope you have a great experience.

Regarding the talks, I recommend you post them to the Talks section of this forum, where more people are likely to see them!