BDD / TDD criticized

General Prog (Archived)
,
45

I still think that Type Driven Design is overall superior to TDD and so forth. By thinking about how the data is represented as it passes through an API is far more important to me than the API itself. Say a function in a module called, oh Html, takes a string and returns a Safe of string (in ocaml parlance) or struct Safe... (in C parlance) is far more descriptive to me than whatever the name is as I can tell pretty well what the function will do based on the types it takes and returns and in fact ‘those’ representations end up driving the API as well as the API becomes transformers from (instanced) types to types only, not a set of calls to ‘do stuff’.

Now of course it is still good to test above that, but most tests are not needed anymore as the type system ‘is’ the test for those.

6 Likes
46

Especially when done correctly.

2 Likes
47

@OvermindDL1 You are correct but you are venturing in the static typing land – Rust, Haskell, OCaml, Go etc. Elixir / Erlang only have limited support for enforcing types and a good chunk of it comes in the form of runtime errors and not compile errors.

What I found to be working very well for me in Elixir though, are roughly these:

  1. Make your function signatures picky. Use when and pattern matching as much as possible. This will help you catch a good percentage of the most common bugs. Also absolutely use @spec and Dialyzer! It takes extra time but gosh, the headaches it saves you from. The extra time spent is 100x worth it.
  2. Use mocking but judiciously. Jose described it perfectly: Mocks and explicit contracts. This way you actually make contracts for your internal API which helps when you misspell a function name or a parameter name in keyword list or map. Thinking about contracts also forces you to think harder about dependencies between your modules which is always good.
  3. Use property-based testing. That way you are not falling into the trap of wishful thinking that the classic unit tests are luring you into.
  4. Have integration tests – namely those that simulate full customer workflows, like visiting the home page, clicking a product, adding to cart, then signing in, then going through all checkout steps, then order, then check how do your internal system move the order through its logical states, etc. to infinity.

I know I am am not saying anything revolutionary or new. Strange though how often even seasoned programmers overlook these.

TDD is basically evangelism and as such it must be mostly avoided, in my opinion.

We the programmers forget we are paid to bring additional value. We forget that all too often.

6 Likes
48

The text you pasted had me laughing and agreeing with it the whole time. Thanks for the share!

49

I would not so sure about GO as is does not have generics.
In mu opinion Go has bad type system.
Check this post :slight_smile:

1 Like
50

DevTernity 2017: Ian Cooper - TDD, Where Did It All Go Wrong

I find this talk isn’t defending TDD/BDD as it is commonly practiced. Yet it goes back to “the sources” (Test Driven Development: By Example (2002), Refactoring: Improving the Design of Existing Code (1999), Refactoring to Patterns (2005)) to discover the actual intent behind the original practices. In my opinion it ends up in a place aligned with Mocks and explicit contracts.

TLDNR: Test the behaviour of the “public API” - not the implementation details

It concludes with:

  • The reason to test is a new behaviour, not a method on a class
  • Write dirty code to get green, then refactor
  • No new tests for refactored internals and privates (methods, classes)
  • Both Develop and Accept against tests written on a port
  • Add Integration test for coverage of ports to adapters
  • Add system tests for end-to-end confidence
  • Don’t mock internals, privates, or adapters

He does emphasize that when tests are used to discover a suitable implementation, those tests will have to be deleted in the end as they are coupled to the implementation details .

He uses the term Duct Tape Programmer quite a bit.

He also references the Fowler, Beck, DHH conversations on Is TDD Dead?.

5 Likes
51


That would be me in a job interview where I was asked to do TDD in a team of “software craftsmen” (credits:
https://twitter.com/iamdevloper/status/1022109902918086661)

1 Like
52

Yes. I find that even in C++, with care, one can use the type system to eliminate most of the trivial bugs. And of course languages like Scala& Rust (Haskell, OCaml etc) are designed to force that rather than merely allow it.

Coplien’s points are interesting.

#1 I think is the strongest point and is definitely what bothers me the most–TDD absolutely focuses on minutiae, from beginning to end. I remember having the corporate “agile coach” visit our team for a day of training. We were presented a problem, toy in scope but with enough subtleties to make it interesting, and of course I immediately started thinking about data representation & API. After we worked on it a bit, TDD was introduced, we were instructed to throw out our work so far and start writing tests, then make them pass. At the end of this the instructor crowed “now, see how different your data structure is”. Well, no, actually, mine wasn’t–a good data design was still good :wink:

If you have a framework designed for a specific domain, and you have a problem in that domain, and the framework is good and fits your problem well, then all you have left to worry about is minutiae. I think that describes a lot of classic RoR applications back in the days before SPA and real-time distributed updates were things. So I think that specific applicability contributed a whole lot to the popularity of TDD.

The problem is, training people in that focus leaves them unable to recognize where the big picture of the framework no longer fits well with the big-picture needs of an app. This is a huge blindspot for developers, leading them to struggle against the design of their tools.

Having said all that, I did find that some training on TDD gave me a new perspective that is useful: thinking about testability when I’m thinking about data types and APIs. It’s not really a different set of criteria, but thinking about it sometimes helps me find an opportunity for decoupling or decomposition which I had missed.

#2 is right, and design by contract is something that really should be taught as part of TDD, but sadly seems not to be–I suspect some of us would object to TDD less if it taught working out the contracts first rather than jumping right into tests aimed scattershot at whatever minutiae of the API we happen to come up with before we really think through the API (or, put another way, Liskov nailed it before most TDD proponents were born and people ought to still be reading Abstraction and Specification in Program Development).

#3 as stated is wrong for 2 reasons, yet still correct in spirit. First TDD does find and correct errors in the application code, second most test errors (in my experience) simply fail to find application errors but not introduce new ones. So it does not double the errors in your running code. However, the point still stands that programmers will commit errors in test code, and this will necessarily limit the effectiveness of TDD to be less than what its evangelists claim.

1 Like
53

I’m loving having an IDE that runs dialyzer in the background and provides me with near real-time error listings. THAT makes it painless to start using @spec!

(VSCode…, but there are plugins for other editors that do it as well…)

2 Likes
54

12 posts were split into a PM: Off-topic posts from TDD thread

56
57

Each of us needs to assess how best to spend our time in order to maximize our results, both in quantity and quality. If people think that spending fifty percent of their time writing tests maximizes their results—okay for them. I’m sure that’s not true for me—I’d rather spend that time thinking about my problem. I’m certain that, for me, this produces better solutions, with fewer defects, than any other use of my time. A bad design with a complete test suite is still a bad design.

Notice that Rich Hickey seems to be speaking as if he’s working solo.

When working solo or with a very small team, perhaps an extensive suite of tests is less crucial than when you’re working on a very large codebase with several teams and many programmers, where no individual knows more than a relatively small part of the codebase.

Having tests might also be useful when you’re working on a project where the programmer turnover is high, where the tests can function as a sort of documentation safety new for new programmers.

1 Like
58

Hickey is saying this in the context of TDD, see http://www.gigamonkeys.com/code-quarterly/2011/rich-hickey/

59

I have seen articles defending TDD with numbers resulting from scientific research. Here’s another paper:

http://people.brunel.ac.uk/~csstmms/FucciEtAl_ESEM2016.pdf

This painstaking study is the latest in a long line to find that test-driven development (TDD)
has little or no impact on development time or code quality. Here, the authors repeated an
earlier study with a couple of new wrinkles, and then blinded their data before giving it to
someone else to analyze to remove any possibility of bias. The result: no significant difference
between TDD and iterative test-last (ITL) development.

60

If you have to blind the data, the signal is weak. Also, a study on 21 grad students does not (should not) really translate to our daily practice (frankly, the longer people stick around universities, the worse their coding skills - I prefer my juniors straight out of high school ;-)).

Bashing TDD/BDD is of course like carpenters bashing a skilsaw. Because it won’t make rabbets or something like that (not that it can’t. You shouldn’t). It’s a tool, not a religion. Skilled craftsmen know when to use the tool and when to avoid it. That is the hard part of software development, and things like “100% test coverage”, “always program in pairs”, “scrum”, etcetera, are either training wheels or hugely situational.

I’ll write code test-first, test-last, no testing at all (especially in “clean” languages like Elixir, I often look at code and conclude it’s “obviously correct” - which I know is a risky proposition ;-)) and I encourage the teams I work with to do the same.Test-first usually happens when the design isn’t clear and I use “London-style” methods to drive the design. Sometimes nothing is clear and I start writing the documentation.

The major take-away is: it all depends. But if you lack the skills to make the decisions and you don’t have anyone to guide you around, I would say that “TDD and try to obtain a very high test coverage” is a very good way to keep you from shooting your feet off. At a cost, of course - see it as an insurance premium, it’s higher for beginner drivers for a reason.

2 Likes
61

That should be proved scientifically, hahaha. Not by high school students please.

Allas, there are many religious followers. And gurus / televangelists (the gurus moralizing the masses on expensive conferences and lateron f.e. youtube) taking profit. Personal hygiene means freeing oneself of religious opinions or at least acknowledging an opinion is what it is (an opinion, not a truth).

A strawman?

Read other research papers where a positive effect is “found”. I think a relevant research question is hard to formulate (what is quality code?) + positive or negative effects of tdd hard to quantify. That should be written about in a research paper.

62

Discussion on HN: https://news.ycombinator.com/item?id=12740456
I like f.e. this comment:

The studies I’ve read all end up showing that principled testing helps,
but test-first and TDD (strict red/green cycle, code only enough to pass
the new test, etc) provide no additional benefit over anything else that
gets the tests written.

The “it depends” always comes from the echo chamber trying to justify
their desire to believe that TDD isn’t completely useless. It actually feels
quite similar to the claims I’ve seen from practitioners that reikei, faith healing,
etc aren’t complete bunk.

63

And what about testing on production ? This days some system are so complicated that that there is only way to test them on production, like using canary deployment and metrics …

In my opinion more useful than tests are looking into metrics. Deploy new build, compare metrics with old build…

1 Like
64

The problem I have with research papers that try to cross psychology/sociology with computer science - the sort of paper that takes a bunch of people and assigns them work and then studies their behaviour and outcomes - invariably fall way short of “real life” and I don’t think that you can extrapolate from artificial coding tasks to the complexity of having 50+ engineers hacking away at a dozen code bases for years. If there’s one truism in our field, it is that our activity is highly social and the rules of complexity science reign supreme - the outcome is highly non-linear.

I’m waiting for a smart professor to talk someone into coughing up the requisite millions in funding to have two companies develop the same thing, one with TDD, one without. I’m sure that the paper will have “more research is needed” somewhere in there.

2 Likes
65

That would absolutely never work as research, though. I don’t see how people think this is something you can realiably research. There are too many variables and you’ll always end up with tons of reasons that could be the real reason you ended up with your result.

Having two companies implement something with or without X/Y isn’t going to prove anything about the merits of X/Y, it’ll simply prove that for that specific task company A/B was better. It’d be a massive waste of some country’s tax money were something like this to be funded and akin to selling snake oil for whichever researcher was trying to convince the private sector about it as well.

Edit:

To put another spin on it: Why does it matter so much and why do people need these obviously never sufficient studies to validate their experience? I have things that I think are so obviously better than the alternative and I think using them makes that obvious enough to anyone with even a minimal amount of experience and ability to reason… But none of that is provable in a realistic way. It could be observable in teams and during projects, but to you’ll never get meaningful results like that.

I would never trust a study on any of these kinds of things, regardless if they agree with my point of view or not. If people are looking for ways to convince other people they should probably look elsewhere, because if someone is convinced by something that doesn’t even hold up to basic scrutiny maybe their agreement isn’t worth much.

2 Likes