What is your testing philosophy?

In the Udemy Elixir Bootcamp course we’re told something along the lines of:

Write doctests as if you were showing someone how your functions work. Write unit tests for behaviours that really matter to you inside that module.

How does everyone feel about this and what is your own philosophy when it comes to testing?

1 Like

After 10 years of developing and writing thousands of tests, I learned that the probability of making an error is quite similar in code that “does really matter” (whatever it means) and does not. I always try to follow this simple rules:

  • make a test to everything you write
  • keep tests as simple as possible (one thing <-> one test, if possible)
  • don’t regret the time spent on tests refactoring and making them better and simpler
  • an evolution of your code is an evolution of your tests
  • 100% coverage is your holy grail :slight_smile:
2 Likes

There has been a thread with some critics: BDD / TDD criticized

2 Likes

I aim to test each relevant context that the application can be in when accessing an important piece of functionality. It’s easy to write a test checking that the functionality succeeds, but it’s often equally important to specify relevant contexts that the functionality will fail in. If this isn’t done, tests may have 100% code coverage but some relevant contexts may still be overlooked. Aim to test less contexts by reducing the amount of responsibility the code being tested has.

Well organised tests are important so that future developers can easily see which contexts have been tested. The test name should reflect what the assertion is doing, and assertions relating to the same piece of functionality (but executing it in different contexts) should be grouped together. In this way, tests double as clear specifications for what the code should and should not do.

1 Like

In my humble opinion, ‘test-driven’-development only works when you know beforehand exactly what you are going to solve, and how you are going to solve it. In practice, this is of course not the case; a project specification is far from constant.

I have worked with teams in the past that had far too rigid tests, which meant that every feature-change needed to be changed both in the code, and in, say, 20+ tests. As the ‘Why most unit testing is a waste’ paper (which the thread @StefanHoutzager linked to discusses in more detail) said: A test that never fails tells you just as little as a test that always fails.

Also, ‘code coverage’ is about as good an indication of how well-tested your application is, as ‘lines of code per day’ is for developer productivity.

What I do think is important is the mindset behind TDD, which is: Think before you code. In practice, I write most tests after I’ve written my code, but I do believe that tests are important. It is a delicate balance: Too little tests, and your application might perform weird at times you do not expect. Too much tests, and your application becomes hard to maintain. Sola dosis facit venenum.

I myself am a great fan of Hammock-Driven Development, because it results in better code even without requiring you to write all tests beforehand (and because hammocks are awesome, of course!):


As for testing in Elixir: I wholeheartedly agree with the statement you quoted above from the Udemy Elixir Bootcamp course. Doctests are great as examples and simple unit/regression tests. When you need to test some edge cases that arise in a more complex context, then these tests can better be written in their own test file.


Another thing: Many people that are new to Elixir think that the “Let it Crash” mentality also means to “Let it burn”. But I think that what we actually usually mean is: We write tests to avoid the simple bugs, but we won’t actively look for the Heisenbugs, because these will find you. It is ridiculously hard to spot a Heisenbug, and instead of wasting too much time on this (and still possibly missing one), it is better to just make you application fault-tolerant: prevent it from burning down even if it crashes.

10 Likes

Well it took me really a while to grok this, but I only can sign what you have learned at udemy.

A doctest is not to test you function via documentation, but it is to find flaws in your documentation after changing the API a bit.

Besides of this, my philosophy of testing is varying :wink:

I really do like to test before I implement, but thats not always possible.

At work we do only test a very small subset of critical functions and we do introduce regression tests as soon as there have been known bugs fixed.

In general I realized that I do test much less when I do have a very strong type system, Often I only have one or to tests that test rough behavior (valid input gives valid output and invalid input will throw an exception, stuff like this) while in languages with limited or no compile time types, I do much more testing and even try to do some randomized stress-tests to uncover hidden edge-cases, very similar to property based testing.

4 Likes

In my humble opinion, ‘test-driven’-development only works when you know
beforehand exactly what you are going to solve, and how you are going to
solve it.

That is just absolutely backwards; TDD is intended to address fluid
and poorly-defined requirements as part of an iterative development
process where refactoring is an accepted regular activity.

And yes, I’ve seen cases of “too many tests” or “100% test coverage”
that obviously came after the fact and do little or nothing to improve
code quality - but that is not TDD.

For anyone who hasn’t read Kent Beck’s seminal book, I recommend it :slight_smile:

YMMV,

1 Like

I haven’t learned the Elixir tools (including doc-tests) well enough to apply this as rigorously as I do in other languages, but generally:

  • When starting on a major piece of functionality, or fixing a bug that doesn’t yet have a test, write the test first. Ideally one that treats it as a black box, testing only the output, or (in languages and frameworks that rely on them) necessary side effects. Some degree of mocking and stubbing is OK, primarily to save “expensive” calls… but if you find yourself having to do a lot of it, caring about the internal implementation, just to get some test to work, something is Very Wrong with the high-level design. Most likely having things (whether they be Elixir modules or Ruby classes or C units of compilation or whatever; henceforth “code-chunk”) too coupled and with too many responsibilities.

  • Any non-trivial subpart that you have to create in the course of making the above work, gets its own test. (And, if it’s not directly in support of the purpose of the containing code-chunk, possibly moved out into a more appropriate one, which may be a new one.) So, you wind up with sort of a “stack” of pending tests. Trivial subparts (that still belong in the same code-chunk) should be made private, if possible.

  • If it’s a new feature that’s visible to the user, make sure you include an end-to-end test. I’ve been burned by features that had tests to ensure the correct result was calculated, but none to ensure that it was displayed, where the display layer either just didn’t show it, or did something that made it incorrect.

  • That’s mainly for the primary happy path. Alternate happy paths, and sad paths, should have their tests written at as low a level as possible (e.g., unit tests on a model rather than end-to-end browser tests). Ideally, the primary happy path test should include that it properly outputs the result of something lower-level, and that is what you can use for the rest of the tests.

2 Likes

The majority of apps I built has been a web interface for a database, mainly CRUD applications. I used to have this assumption that unit tests are great and you need to unit test all the things; that you test for the sake of testing.

Except when it just doesn’t make sense.

CRUD apps tend to just be a glue for other components. Most of the time, there won’t be much logic involved. So you write a unit test for a service that constructs and runs your query by mocking the DB? You essentially are testing the query generator library, and the authors have tested that for you. The information you need is whether you glued the lib correctly, which means integration tests.

Where unit tests make sense in CRUD apps is for your controllers, where it might have branches depending on the params. You need to mock out the DB service for that. But other than that it would be mostly glue code and you’ll be writing e2e tests by spinning up the server and hitting the endpoints with test requests.

So the base of my philosophy is: design your test based on what information you would like to get. There are several articles which inspires me for this, like Trustworthy Unit Tests – Testing the Data Access Layer and Writing Tests for Data Access Code – Unit Tests Are Waste.

Of course, you would need unit tests if your code includes complicated algorithms with a set of inputs generating possible outputs.

All that, but I’m still somewhat noob in testing, so you might want to take my opinion with a grain of salt.

1 Like

Entirely this!

E…yeah… I do admit that testing almost vanishes when I use a language with a decent type system (especially type systems that can encode extra attributes other than just storage, like C++ templates and ML languages can do). >.>

But in Elixir I often write code if I already know what I want, if I do not then I write tests to flesh out the API first of what I want, then I write the code to fulfill it (this is more common, but not ubiquitous).

3 Likes

For me:

  • Unit Tests
    • Good for small, self-contained algorithmic functions, especially when enumerating multiple input permutations.
    • Fit very nicely with doctests, e.g. in my IbGib.Helper module.
    • TDD is very well suited for this, but not a hard and fast rule.
  • Integration Tests
    • What I use most of the time.
    • Can test primary workflows (i.e. non-edge cases)
    • Can act as examples of how to use a library.
    • Often use TDD because it’s a great guide, but again, not a hard and fast rule (especially for more experimental approaches).
  • Regression Tests
    • If I find any decent, non-trivial bug, always do TDD to clearly define expected behavior.
    • Don’t want to ever have a bug happen twice.
    • Any decent number of bugs in a given area indicates that more tests are needed, and possibly an architecture/design rethink.
5 Likes