I placed this and the previous copy/paste from Lamport in the context of this thread, where the subject is "BDD / TDD criticized’. Of course the scope of Lamports articles is broader. But it’s all about what should go before coding (and that is not writing tests but thinking/writing/designing “above” the code level). Call it big design upfont. 
Btw Donald Knuth in an interview http://www.informit.com/articles/article.aspx?p=1193856
But that is no more than an opinion of a famous computer scientist. Lamport provides substantiation.
2 Likes
That’s exactly the average environment of most commercial programmers though. Treading carefully in a team setting inside a complex project – via unit tests – is a basic common sense, not a fashion trend or cargo culting. It helps you to not break other people’s work and to progress your backlog tickets.
Agreed. 
When you are building something by yourself then you have a bigger freedom overall then it’s really pointless to tread as if you are in a minefield. Just create and experiment! The time for automated validation tests usually introduces itself when the time comes – because at certain point you have too many moving parts and you need the extra peace of mind that you can progress with confidence.
(I imagine that’s like building your own mini-car in the garage and starting to gradually tighten the nuts and bolts once it starts getting into workable shape and you are ready to drive it for a test run.)
To me TDD is a badly overdone religion. It makes perfect sense in a particular set of circumstances but it’s very far from that universal truth it’s sold as.
Vehemently disagree. Besides common sense is not “what is right” but “what most do”. A Dutch saying: “Whose bread you eat, whose word you speak”. Stems somewhere from the bible.
1 Like
Can’t edit my reaction anymore. Add: for extensive reasoning behind my disagreement see for example the very start of this thread, with a link to “why most unit testing is a waste”.
https://twitter.com/HenrikJoreteg/status/1098829955822018562
https://twitter.com/dan_abramov/status/1098560575712432128
Dan Abramov @ dan_abramov 19 uur19 uur geleden
In particular I’m curious about the emotional investment into a particular testing methodology. Could it be because they’re all not that great, and so we develop devotion in order to feel our choice is justified?
Becca Nelson Bailey @ beccaliz 18 uur18 uur geleden
I think many of us have been told that good testing practices are what separates “good” programmers from “bad” programmers. We have given testing an almost moral value, but we don’t have an absolute standard to judge by, so the standard becomes our own practices.
1 Like
A plea to do testing:
It took about five years until I understood how testing works (for me). And I can’t imagine doing any programming without anymore. My key to success with testing was to not see it as something I do in addition to my code but just as part of it. I write tests first because I first wanna define how my not yet written code can be used. If it is hard to think about a simple test for a new function, I first refactor my existing code thus I can come up with a simple test. Because if it’s hard to write a test for something, most likely that stuff will be hard to understand and use for other programmers (or myself in the future).
So, the key point here is, not even think about how I will implement the function before I have a definition of how to use it then. For example, if I have in mind that I will use a struct for something, tests often drive me to use a simple list.
When I have to continue on the code I haven’t touched for a while, I first run
mix test --trace --seed=0 and immediately have a specification of the status quo.
When I look out for something on Github, I first read the tests to understand how I can use and eventually change/extend that code. If there are no tests, or in case I can’t understand the tests, I skip that repository.
Other than that, of course, I feel so comfortable when there is a trustworthy test suite if it comes to refactoring existing code.
I gave up arguing about TDD, I just do it and enjoy a much stressless life.
A plea for requirements engineering before coding: BDD / TDD criticized. Lamport makes the distinction with TDD clear in his paper.
1 Like
As I wrote, testing is part of my way of developing applications. This doesn’t mean that I don’t do requirements engineering up front. Just the opposite is true. But, when I start coding, I start with a test.
Often the tests / the fact it is hard to find a proper test / shows me when my upfront design has flaws in details. And in this case, I tend to rework my blueprint rather than question if I’ve waste time writing the tests.
Start without a blueprint is wrong. Do it without testing is wrong. Testing just for the sake of coverage is wrong. Strictly following a blueprint for any price is wrong. Doing it right needs a lot of practicing and failing is more likely than success.
So, I totally support your plea for requirements engineering but at the same time, I try to encourage people to do proper testing.
Can I hope that we can agree on, that one is just as important as the other?
This can happen when you’re finding out what you really want. Testing shows “how sloppy your thinking is”. Happens to me also, by far the most requirements I find out without testing however. Often my tests are end to end, as they give the most ROI. I am going to put a small js thing on github in a short while. The code contains an extensive list of requirements and as far as needed comments in the code explaining the how. Not an extensive testsuite. The code should be understandable by reading requirements, the code itself and the comments in the code. Tests would not add value with respect to understandability I think. I would accept and maybe even write tests myself per requirement if it would not hamper refactorings, and if it did not cost so much extra time. I would not trust a testsuite that much as the guarantees given by the results are not quite 100%.
Discussion on twitter with Ron Jeffries etc.
Use of Formal Methods at Amazon Web Services, paper from amazon:
Kent Beck:
tdd - How deep are your unit tests? - Stack Overflow
Have fun reading the reactions on HN 
1 Like
This psychological concept of intellectualization is recognizable in quite some more discussions of course.
2 Likes
From
https://blog.usejournal.com/lean-testing-or-why-unit-tests-are-worse-than-you-think-b6500139a009
Found the comments far more interesting than the article - which for some reason felt to me like suggesting that straightening one’s path is the solution to solve this problem:
¯\ _(ツ) _/¯
Someone also linked to this: Just Say No to More End-to-End Tests (2015)
I do not understand what you mean. I found the article good in that it values return of investment and sees the least roi in unit testing, generalized speaking. Like Coplien (see the very first message in this thread with the link to his http://rbcs-us.com/documents/Why-Most-Unit-Testing-is-Waste.pdf ), like what Rich Hickey says in that first message also. Concerning the link you send: I read more than one time about the “pro unit-test” culture at google. Not every company has the same culture, for example facebook has a totally different testing strategy. Here someone from google looking for another company to work for and interviewing them about their testing strategy (expect not much about those other strategies, you can find something about f.e. facebook easy online): https://medium.com/@moishel/unit-tests-arent-free-c76e4bdf5332
Director of engineering at facebook (and not quite a dummy) Erik Meijer says
in that it values return of investment and sees the least roi in unit testing.
To me the yardstick the article applies seems to follow thinking along these points:
- Tests are a byproduct of arriving at the end product. The end product has all the value, tests have none of it.
Harsh but true.
- Tests contribute no direct value, so their cost need to be minimized.
This is where things get slippery, it is easy to simply measure money spent - it is much more difficult to assess if the money is being spent in the right place, i.e. effectively.
For example not writing unit test saves money but what about the cost of time spent debugging, just trying to find the root cause of a defect that was uncovered by end-to-end testing? Granted creating unit tests at too low a level of granularity can be expensive but having something in place that can narrow the search or better yet catch it before it gets to e2e can also save money. I’m not convinced that e2e-and-fix is always and consistently cheaper.
- e2e tests are closest to running in production, therefore it is self evident that they are the most valuable tests.
Nobody is arguing that e2e isn’t necessary. But even e2e can miss part of a system’s operational profile and for greenfield systems the operational profile is often just a guess. So unit testing is part of a “defense in depth” strategy - try to catch problems as early as possible and have multiple points where problems could be detected if one of the lines of defense fails. But any type of redundancy incurs cost, the issue is whether it is actually worth it.
The other aspect that seems to get no consideration is the economic risk/cost of a defect making it out into the wild. Not all defects are created equal. I wouldn’t consider most of the services of Netflix, Facebook or Google as mission critical. Big deal if somebody can’t get movies for a few hours or even a few days - even if they loose a few customers, they have so many of them - and even accidentally exposing a few thousand credit card numbers seems to be survivable for them. Other organizations may be considerably more vulnerable to the effects of defect exposure.
- e2e is the most valuable so that is the only testing that should be done
- Hence e2e has the best ROI
I do not understand what you mean
Implicitly the article suggests that unit tests are unnecessary detours that cost money therefore the solution is to stop taking detours and move straight to the goal of the working, successful system - which retroactively informs what the complete set of e2e tests should be (for due diligence).
NATO Software Engineering Conference 1968, page 21:
Kinslow: The design process is an iterative one.
Why is software development iterative? I think because while we’re still at point A we may think we know where point B is but usually we’re wrong. So often the only way to progress, is to move forward a bit and then re-assess with any new information that has been gathered whether we are moving in the right direction. That type of progress rarely moves in a straight line. Big Design Up Front tried discover the straight line to B - up front - while staying mostly at A but then often failed to acquire the knowledge where B actually is or that they actually needed to go to C.
Also unit testing serves a somewhat broader objective than e2e testing. Unit testing is also supposed to inform design (usually to enable refactoring to reduce volatility of the marginal cost for the next feature). But like any tool, unit testing can be underused, overused, misused and abused.
It also doesn’t help that there doesn’t seem to be a universally shared understanding of what unit testing actually is (my interpretation was largely informed by Working Effectively with Legacy Code (2004) - i.e. building systems that are easier to change in the face of continually changing business requirements).
like what Rich Hickey says
https://twitter.com/richhickey/status/119010218670694400
He prefers to rely more on thinking (Hammock Driven Development) and describes TDD as “banging into guard rails”. Though doing anything in a mindless manner is rarely productive …
3 Likes
Not harsh in everyone’s (me f.e.) perception. While flying to the moon a rocket throws off parts that are no more needed, but they were necessary to arrive at the moon. The dumping was necessary also.
I have no formal proofs, and indeed you will have to spend more time finding the rootcause of a defect compared to a unit test, but in general I found writing e2e tests much more valuable. E2e and unit tests test different things. An e2e test tests real user scenario’s and integration of components. Unit tests can be used as complement as you and others have explained. Ok, we know all that. A nice extra is that e2e tests can be used with legacy code that is not neat. The problem I see is what Martin Sustrik calls “Unit Test Fetish”
http://250bpm.com/blog:40
The far cry of Uncle Bob: “if you don’t do TDD, you are unprofessional”. All those filling up of public repositories with unit tests because when not they might be taken for unprofessional. Not throwing away tests that were only of value during initial development / while learning the language. Not getting / wanting a job because of Uncle Bob standing at the door again.
You have a point here. Certainly with parts of safety critical software you have to test rigorously (formal verification) before releasing to production.
My idea. Anyone reading this doing shift-right testing btw? Any stories to share?
2 Likes
