BDD / TDD criticized

,

Each of us needs to assess how best to spend our time in order to maximize our results, both in quantity and quality. If people think that spending fifty percent of their time writing tests maximizes their results—okay for them. I’m sure that’s not true for meI’d rather spend that time thinking about my problem. I’m certain that, for me, this produces better solutions, with fewer defects, than any other use of my time. A bad design with a complete test suite is still a bad design.

Notice that Rich Hickey seems to be speaking as if he’s working solo.

When working solo or with a very small team, perhaps an extensive suite of tests is less crucial than when you’re working on a very large codebase with several teams and many programmers, where no individual knows more than a relatively small part of the codebase.

Having tests might also be useful when you’re working on a project where the programmer turnover is high, where the tests can function as a sort of documentation safety new for new programmers.

1 Like

Hickey is saying this in the context of TDD, see http://www.gigamonkeys.com/code-quarterly/2011/rich-hickey/

I have seen articles defending TDD with numbers resulting from scientific research. Here’s another paper:

http://people.brunel.ac.uk/~csstmms/FucciEtAl_ESEM2016.pdf

This painstaking study is the latest in a long line to find that test-driven development (TDD)
has little or no impact on development time or code quality. Here, the authors repeated an
earlier study with a couple of new wrinkles, and then blinded their data before giving it to
someone else to analyze to remove any possibility of bias. The result: no significant difference
between TDD and iterative test-last (ITL) development.

If you have to blind the data, the signal is weak. Also, a study on 21 grad students does not (should not) really translate to our daily practice (frankly, the longer people stick around universities, the worse their coding skills - I prefer my juniors straight out of high school ;-)).

Bashing TDD/BDD is of course like carpenters bashing a skilsaw. Because it won’t make rabbets or something like that (not that it can’t. You shouldn’t). It’s a tool, not a religion. Skilled craftsmen know when to use the tool and when to avoid it. That is the hard part of software development, and things like “100% test coverage”, “always program in pairs”, “scrum”, etcetera, are either training wheels or hugely situational.

I’ll write code test-first, test-last, no testing at all (especially in “clean” languages like Elixir, I often look at code and conclude it’s “obviously correct” - which I know is a risky proposition ;-)) and I encourage the teams I work with to do the same.Test-first usually happens when the design isn’t clear and I use “London-style” methods to drive the design. Sometimes nothing is clear and I start writing the documentation.

The major take-away is: it all depends. But if you lack the skills to make the decisions and you don’t have anyone to guide you around, I would say that “TDD and try to obtain a very high test coverage” is a very good way to keep you from shooting your feet off. At a cost, of course - see it as an insurance premium, it’s higher for beginner drivers for a reason.

2 Likes

That should be proved scientifically, hahaha. Not by high school students please.

Allas, there are many religious followers. And gurus / televangelists (the gurus moralizing the masses on expensive conferences and lateron f.e. youtube) taking profit. Personal hygiene means freeing oneself of religious opinions or at least acknowledging an opinion is what it is (an opinion, not a truth).

A strawman?

Read other research papers where a positive effect is “found”. I think a relevant research question is hard to formulate (what is quality code?) + positive or negative effects of tdd hard to quantify. That should be written about in a research paper.

Discussion on HN: An External Replication on the Effects of Test-driven Development [pdf] | Hacker News
I like f.e. this comment:

The studies I’ve read all end up showing that principled testing helps,
but test-first and TDD (strict red/green cycle, code only enough to pass
the new test, etc) provide no additional benefit over anything else that
gets the tests written.

The “it depends” always comes from the echo chamber trying to justify
their desire to believe that TDD isn’t completely useless. It actually feels
quite similar to the claims I’ve seen from practitioners that reikei, faith healing,
etc aren’t complete bunk.

And what about testing on production ? This days some system are so complicated that that there is only way to test them on production, like using canary deployment and metrics …

In my opinion more useful than tests are looking into metrics. Deploy new build, compare metrics with old build…

1 Like

The problem I have with research papers that try to cross psychology/sociology with computer science - the sort of paper that takes a bunch of people and assigns them work and then studies their behaviour and outcomes - invariably fall way short of “real life” and I don’t think that you can extrapolate from artificial coding tasks to the complexity of having 50+ engineers hacking away at a dozen code bases for years. If there’s one truism in our field, it is that our activity is highly social and the rules of complexity science reign supreme - the outcome is highly non-linear.

I’m waiting for a smart professor to talk someone into coughing up the requisite millions in funding to have two companies develop the same thing, one with TDD, one without. I’m sure that the paper will have “more research is needed” somewhere in there.

2 Likes

That would absolutely never work as research, though. I don’t see how people think this is something you can realiably research. There are too many variables and you’ll always end up with tons of reasons that could be the real reason you ended up with your result.

Having two companies implement something with or without X/Y isn’t going to prove anything about the merits of X/Y, it’ll simply prove that for that specific task company A/B was better. It’d be a massive waste of some country’s tax money were something like this to be funded and akin to selling snake oil for whichever researcher was trying to convince the private sector about it as well.

Edit:

To put another spin on it: Why does it matter so much and why do people need these obviously never sufficient studies to validate their experience? I have things that I think are so obviously better than the alternative and I think using them makes that obvious enough to anyone with even a minimal amount of experience and ability to reason… But none of that is provable in a realistic way. It could be observable in teams and during projects, but to you’ll never get meaningful results like that.

I would never trust a study on any of these kinds of things, regardless if they agree with my point of view or not. If people are looking for ways to convince other people they should probably look elsewhere, because if someone is convinced by something that doesn’t even hold up to basic scrutiny maybe their agreement isn’t worth much.

2 Likes

What I got from the original statement is that design is paramount. I have been programming for 34 years and agree. I am also in the write tests after camp.

To be honest, I also don’t give a fig about design patterns. Thinking in design patterns, starting development with tests, and even designing with a specific language in mind all affect the design negatively, in my opinion. Come up with the best design you can to solve your problem, then choose the tool or tools best suited to implement your design. Never let the tail wag the dog.

4 Likes

We have replaced fact-based data-backed attitude with a faith-based
wishy-washy peace-hug-freedom hippie agile way, forcing us mechanically
to follow some steps and believe that it will be good for us. Agile has taken
us a long way from where we started at the turn of the century, but there are
problems. From personal experience, I see no difference in the quality of
developers who do TDD and do not. And to be frank, I actually see negative
effect, people who do TDD do not fully think hard about the consequence of
the code they write - I know this could be inflammatory but hand on heart, that
is my experience. I think TDD and agile has given us a safety net that as a
tightrope walker, instead of focusing on our walking technique, we improve the
safety net. As long as we do the motions, we are safe. Unit tests, coverage,
planning poker, retrospective, definition of done, Story, task, creating tickets,
moving tickets. How many bad programmers have you seen that are masters of agile?

There are developers who name themselves “software craftsmen”, especially among those specialized in ruby / elixir. That has historical reasons explained elsewhere. Not that it is the main reason for starting this thread ;-), but I have even been rejected for a job by such a proud
craftsman because of that horrible thinking for myself.

2 Likes

Indeed. I was very firmly planting my tongue in my cheek when I was writing this. Should have added an appropriate emoji :wink:

As a guy who titles himself like that, I see no relation between the title and the behaviour you describe. There are douchebags with all sorts of titles and in all areas of life. :smiley:

People rejecting other people for reasons not related to the job at hand is sadly happening way too often. And escape hatch terms like “cultural fit” are now used way too often, to the point of abuse. There is such a thing as good or poor cultural fits but most of the time people are just fine working together, provided they follow a basic set of professional courtesy codex.

Fully agreed with only one exception – functional languages just sort of work better for the human brain. We are not good at doing whack-a-mole with 5000 holes like most imperative / OOP languages require us to.

Having the FP fundamentals (like immutability and share-nothing concurrency) helps us concentrate on less things at a time, and doing them better. In that regard, I think that most designs should immediately start with the premise that they need an FP language (unless we are talking NASA or gaming servers).

Outside of that clarification however, you are 101% on point. I understand that companies like SAP have thousands of employees and they need to enforce some standards but IMO that’s doing them disservice.

1 Like

That is true. Lab tests require controlled conditions which is exactly the opposite of what we have in the real life jobs. Only that premise alone invalidates any conclusions they might claim to have extracted.

Yep. Also people find the most creative of ways to cheat in competitions. We just operate differently in an every day job and at a competition. It cannot be avoided.

This is why I stopped arguing with people on HackerNews. Whatever you might say you keep hearing “source?” or “any studies to prove?” – like 19 out of 20 times. How do you explain to these people that the modern science is still uncapable of proving certain phenomena due to its methodology and the human condition itself?

As you said, there are many things people “just know” and they work pretty well – we have those in agriculture, programming, cooking, raising kids, basically everywhere. Yet if we ask science, all these things are “urban legends” and “have no citings or proof they work”. Oh well then.

Maybe the name (“software craftsman”) is not that important, but what it stands for, at least in my view, does. In the craftsmen manifesto tdd is not mentioned. But in the book “The Software Craftsman: Professionalism, Pragmatism, Pride” you can find paragraphs on f.e. “How Do I Convince My Team to Do TDD?”. Do a “software craftsman advanced” course and you get “Best practices: using testing and test driven development […]” Clean Code: Software Craftsman Advanced Instructor-Led Training Course | ProTech.
Uncle Bob, writer of the clean coder and long time proponent of tdd just said

https://twitter.com/unclebobmartin/status/1030819434510745601

The disagreement is left implicit which makes this a strange read, but to me his view (see complete tweet) is new. In the past he said quite something else, f.e. in the clean coder book. Something like “if you do not practise tdd you’re unprofessional”. Opinions like these still sing around. Some employers demand you keep singing this song with them.
Dan North made some valuable remarks on software craftmanship btw:

Software Craftsmanship risks putting the software at the centre rather than the benefit
the software is supposed to deliver, mostly because we are romantics with big egos.

https://dannorth.net/2011/01/11/programming-is-not-a-craft/comment-page-1/

1 Like

We’re encroaching on the territory of philosophy of science here, but science doesn’t need to “prove”, just demonstrate – Math needs to produce proofs. And I think it has been adequately demonstrated how complexity works, which explains a lot of stuff around our practice. It also explains why studies that compare two small groups of people in a lab setting have little value :wink:

So did you, probably. I know I did. But we’re not under a magnifying glass because we wrote books that everybody has on their bookshelves, I guess. I’m super happy that people like Uncle Bob continue to evolve. If anything, the thing that I learned from people of this calibre is that they are mostly just insatiably curious and primarily want to learn, not hang on to their old beliefs. Which makes it harder to keep up to date with what these thought leaders think, but that’s on us; not on them :slight_smile:

2 Likes

Minutiae. :stuck_out_tongue: I agree though.

My point was that many concepts are extremely hard to prove mathematically yet many of us use them daily. You would think that would tell scientists something but nope.

Of all the scientists though, I mostly like mathematicians. They are quite brutal to themselves and many are open to admit they have been doing something wrong for 20 years if new and solid data emerges. Theirs is the right mindset and I enjoy communicating with such people, no matter what they make their money with.

And they are the only ones that managed to get to some sort of real-life answers through game theory and chaos theory.

1 Like

Well stated. I would say this in response: I try to avoid thinking in a specific language or even language type while I am designing, but since discovering Elixir and FBP, I have found that these tools allow me the easiest path towards successful implementation. Truly a pleasure to program in.

1 Like

This idea has riscs. We need to live with lots of unproved assumptions, otherwise we just cannot. But science can help in getting wiser. Knowledge is acquired while for example, well, doing scientific research, but also while f.e. reading this. Piaget had a theory about cognitive development that can shed some light: Cognitive Disequilibrium | SpringerLink (there should be a better link somewhere, but this as intro). There are different reactions possible on the stated psychological phase of cognitive disequilibrium, we may become wiser but other reactions are possible also. Some of those are subject for research by psychiatry. Also a science, sorry. :wink:

Disequilibrium is often an uncomfortable state for individuals, thus we seek to quickly
return to a state of equilibrium.

1 Like