Code inspections vs unit tests

The purposes have an overlap in that both serve the purpose of checking correctness.

I don’t get this. Automated tests are full of assertions. If you mean the safetynet that you construct:

you do inspections after changes also. Unit tests have a cost when changing code. You have to maintain them. Functionality changes, functions are removed and created, names of functions change, their signature changes.

I like to work fact-based, empirical research like

shows reviews to be more effective than testing. See 3.5 Review versus testing and 4.1 Cost and benefits

Claims by Uncle Bob and other agilistas about unit testing are not scientifically proved, for me they belong to the realm of pseudoscience. Moral imperatives (f.e. from Uncle Bob “if you don’t do TDD, you are unprofessional”) are unnecessary, and harmfull. I cannot take them serious.

An example of a highly ranked scientist and programmer that is not enthousiast about safetynets:

For more see the interview https://www.informit.com/articles/article.aspx?p=1193856

Read this thread BDD / TDD criticized by the way?

1 Like

Well that’s exactly the problem – they shouldn’t be compared because they are two different tools serving different purposes.

It’s like being forced to choose between a wrench and a hammer, you know?

1 Like

Nothing that relates to management of people is scientifically proven, as far as I know. Management involves psychology and working with different kinds of character traits in people – and that simply cannot be generalised and put in a comprehensive theory by science in its current form. Nor there are any large-scale experiments that prove anything beyond any doubt.

Bringing science to discussions about management is fruitless. When we discuss it, we are basically all alchemists arguing which exact ingredients will yield gold bars when mixed in the right pot.

I think it’s important we all recognise that. These “papers” you quote have a number of omissions when contrasted with the pure scientific method (like having large sample sizes, asking people from many different areas, analysing responses based on multiple factors, actively working to remove bias etc.). At best, they are from people who claim to have an epiphany after a long career. And they have a bunch of pals in a few universities that pat them on the back and sign their work without much scrutiny. Boom, the paper is now “credible”. That’s all there is to it and let’s please stop pretending otherwise.

So in this regard, your approach of citing very sketchy sources is ill-placed and won’t help with any discussion you might be willing to ignite.

2 Likes

As I said: “The purposes have an overlap in that both serve the purpose of checking correctness.”

This is about unit testing, not about management.

I respectfully disagree.

As I wrote, I think the main purpose of automated tests is not, in fact, checking correctness, but rather minimizing the cost of changes. A poorly-tested project is hard to evolve, as everything you change can break something else.

Code inspections and quality assurance are for checking correctness. A project where no code inspections or quality assurance is performed is more prone to contain defects.

I therefore think that comparing code inspections with tests is comparing apples with oranges: both are necessary, for different purposes, and they are not interchangeable.

A large extent of the paper you quoted is based on anecdotal evidence and subjective interpretation by the author. To me it sounds a bit as if someone was saying: “Seatbelts and headlights were compared as a mean to reduce car accidents in low visibility. Keeping headlights on proved more effective, so they can replace the need for seat belts. Also, seat belts give users a sense of false security.”.

The testing practices described in the article are quite far from agile: in many passages, it sounds like the subject of the article performs testing as an afterthought, possibly executed by different people, as opposed to as part of the development process. For example, it advocates for testing after the code inspection is performed. I intend testing as an integral part of development, and performed by the developer themselves (no matter if one practices TDD or not, TDD is not the point here).

I do think that agile practitioners sometimes go overboard, that sometimes agile is deployed as a mean of controlling rather than as a good software practice, and that in some contexts it takes on the traits of a cult rather than a practice. I also believe that the principles of iterative development and testing as part of the development process are still valuable.

2 Likes

If it was just about asserting correctness and detecting bugs, a round of manual testing would do just as well. You might happen to catch some bugs by running your automated tests, but you won’t catch what you didn’t design the tests for. Simplifying, you cannot reveal a bug that you don’t know of, and if you knew about it you could just fix it.

The value of automated tests is that, once written, they become part of your test suite, pinning down the invariants that should remain even when the implementation details are changed. Every time someone changes some code, the test suite informs them if some of those invariant was inadvertently broken.

The biggest cost of writing tests is payed only once, but you can run the same test as many time as you want during the development process. Therefore, they are efficient to safeguard against breaking functionality while evolving your project.

Inspections and manual testing have a cost every time they are performed. They are a better way to assert correctness and quality of newly added features, but re-inspecting or manually testing everything upon every change would be inefficient.

1 Like

I called it the main goal for many. When you repeat the tests that goal stays the same. For the rest: I know how it works and what the beliefs are.

But it is different. Let me make a silly example. Let’s say I am tasked with writing a function that adds two numbers. I end up writing this:

def sum(a, 0) do
  a
end

def sum(a, b) do
  sum(a + 1, b - 1)
end

I then write a test for it:

test "it adds two numbers" do
  assert sum(2, 3) == 5
end

I am not writing this test thinking “let’s see if my code is correct or not”. I know (or at least I am positive) that this test will pass. I write it to record in the test suite what my function should do, no matter the implementation. This test is not about correctness, it is there to prevent regressions.

A code review or quality assurance check then finds that my solution breaks if one tries to sum negative numbers. It reveals that my code is not correct, and also points out that it is cumbersome and inefficient. My test did not reveal this bug, nor the fact that the code is overly complex: does it mean it is useless?

I then discover the existence of + and change the implementation to:

def sum(a, b) do
  a + b
end

I can run the previous test to make sure that it still passes. I am addressing a different concern (support for negative numbers), but I still want the previous behavior to be supported (positive numbers). The previous test is useful, precisely because I can re-run it at no cost, to make sure my new code still does well what the old code was doing successfully. I will in fact also write another test to protect from regressions regarding negative numbers:

test "it adds negative numbers" do
  assert sum(2, -3) == -1
  assert sum(-2, 3) == 1
  assert sum(-2, -3) == -5
end

I really appreciate your views on tests and agile and pretty much the software culture as it is right now, but I absolutely disagree with you in almost point you make.
This is probably a great example of how there is no absolute right way to build a house, but you can disagree in the details.

It sounds like you are suggesting that unit tests are worthless, and should always be disregarded in favor of people using the software. I think this is wrong, and I will make my case further down. I have written software with and without unit testing, and I know the pain tight unit testing can inflict on the overall developing process, so I do not agree with dogmatic TDD approach. But I also don’t believe that is how people practice TDD.
Furthermore, you mentioned Uncle Bob - he has been a person I used to look up to, because what he said made sense to me, and following some of the stuff he preached made me feel better as a developer and still - I agree, he is a dogmatic “do never change my gospel” kind of person, and some of the more recent things he said are even more disgusting. But you can not disprove things by quoting people who might have done wrong or not, that is an “argumentum ad hominem”, and known as a fallacious argument.
You also quote passages of studies that support your argument, but disregard other aspects of the argument that are way outside of the study, which is another fallacious way of arguing your point you keep using.
I realize this might come off as a personal attack, but it’s not. I just think you are wrong, and you will probably think I am wrong. That’s healthy, and that’s how it should be! You may correct me in any of the points I want to make, if I misrepresent your opinion.

So now to my actual points regarding Code Inspection VS Unit Tests.

1. Unit tests constrict development, and make changes harder
I enjoy writing unit tests, because I like watching red turn to green as a feedback for my efforts. I agree that unit tests can distract you from the actual goal, and in a test suit that was not done with unit tests in mind, the test setup can be far too complicated, just to prove that - indeed - plus(1, 3) returns 4 (if you have unit tests that basically just test usage of basic implementations like this, or implementations that have been tested in a library, delete them.)
Maybe you could say that I am an advocate of deleting tests after writing them. If you want to make sure 1 + 3 makes 4 in your setup… Sure go ahead! But don’t expect anyone (including yourself) to maintain that test, at some point someone will raise the question whether or not you should even have such a function.

I am doing different approaches to software development, I work with 0% test coverage, I work with close to 100% test coverage, I work with code that had tests written after the code, and I work with test written before. I almost certainly do not have your level of experience (sorry for making deductions based on your avatar), but I can certainly say that I am thankful for everyone who bothered to write tests, even if the tests are crap whenever I have to dive into unknown code.
Because their tests describe their thought process. I make a change, suddenly 50 tests are red… Oh oh, I did an oopsie? No, they just wrote all their tests assuming that the arguments will always be exactly X. Ok. Bad assumption, and I have to rewrite the tests anyways to fit my change. And suddenly there is only 1 red test, and I realize that I would break an existing feature with what I did.
I lost some time, but I gained a lot of insight of how the person who wrote this code thought about the process. It can be super frustrating - WHY WOULD YOU - mea culpa - we are all very guilty of writing crappy code from time to time.

However, working my way down from the top level function, to the more nitty-gritty ones, and having tests supporting my assumptions, helps me think about the over-all design, and it helps me rethink! This is my opinion, and it certainly works different for different people, but whenever I break down a function into its components, and I realize that the setup for even one of those unit tests is extremely difficult, then I might have gone into the wrong direction…
This is imo what doctests in Elixir promote. If you can not test it in a doctest, do you really want to do it like this? Should this not be a top-level function? I can not count the number of times I realized that a function should be top level, and thus tested in a more integration way, than unit way, … I can’t count it.

I said that I would advocate for deleting unit tests after writing them, and I think that is a very valid approach. If you are like me, you would want some very small scale verification on parts of your code working. If you are like me, the process of writing

test "it does exactly X if given Y"

helps you think about what your function should and should NOT be doing, and improve code quality. But it might not be beneficial to make code future proof, so by all means… Get rid of development tests! Tests are lines of code that have to be maintained, less code often means less maintenance - yes!

But by no means I think unit tests are a waste of time or could be replaced by pure integration testing, be it by humans or a machine. What you are suggesting is that libraries should be tested by their users, contributors be damned.

2. Code Inspection makes Unit Tests obsolete

*Dislcaimer: I assume we are talking about code inspection as defined by Wikipedia If I am wrong here, I am looking forward to reading what your definition is.

Not only does this disregard my point about how unit tests can help you thinking about code, it also suggests that everyone can afford code inspection.

What if you are a startup, and you think your code could be improved by the masses! You make a library public, and boom, there is your code inspection, right!!!
No, that is not how it works… People will hesitate to contribute to libraries they don’t fully understand, tests will give them security, etc. But I understand you are talking more about specific projects, where everyone is working for a single goal to achieve! Like, that company will never have people leaving, people joining!
Think about a developer who has to do a simple task, in a big project. Would it make them feel more confident about their code if you told them “we will know if it worked or not once we have the Code Inspection results”, or if they could run a test suit that either tells them “all green” or “10 red”?
If you think just doing things in a project that is completely foreign to you, without a tight test frame is completely fine, then you are a hypocrite. I have never met a junior developer, or a senior, who was fine with just changing stuff to what they thought was right, without getting feedback from anyone - or in the case I am trying to make - something.

3. Developers save time by not writing unit tests
This is a very, very controversial point. I have lost hours and hours just fixing tests that did not meet the initial expectations anymore, but I - as in, me personally, people are different - prefer delving into how things USED to be and potentially find bugs before they happen, over months of calm sea and then facing the ultimate storm.
Maybe you are not involved with these kind of things, but web developers have to face the ugly truth of no version can last forever. Upgrading an Ubuntu server to the latest version might feel great, but how long until the app has to be updated? Are you running an Elixir 1.6 app? That is so outdated… Have fun updating to 1.10 (disclaimer, I found Elixir apps to be super easy to update, since I prefer to pull in the least amount of dependencies, and Elixir tends to give very descriptive warnings, but boy… there are some tricky bastards to tackle already!)

A good test suit can give great developers a great peace of mind - (at least I did not break anything.)

4. TDD takes the fun out of development

Now this one is entirely me, putting words into @StefanHoutzager 's mouth. It always sounds like you are saying writing tests first takes away from the developer experience and freedom, basically just testing it manually and throwing the result at users is test enough. Sorry if I misinterpreted you, feel free to correct me.

I confess, I do not always write tests first. I confess I think testing certain template responses in Phoenix (or whatever framework) is a waste of time. I confess I think Uncle Bob is full of ***. I confess I think some of the projects I know like the back of my hands should quit bothering me with implementation details, badly written tests and just let me deploy what I (at that certain time think) is the fix for a bug!

But I generally enjoy the red -> green cycle of tests. I enjoy writing stuff that I know others will look at, my test descriptions, and I enjoy their feedback. I am guilty of writing tests like “it works with X”. I am guilty of overtesting. I am guilty of not testing enough.
I usually am more thankful for my over-testing past self though. At least it forces me to understand why a test was failing!

5. What are you proposing then…?
My approach would raise eyebrows in every community. Me, my next year me, would probably scold me for what I am doing right now.
I write tests… I do your approach from time to time (without the million dollar company you seem to have behind you, or whatever it is that lends you your ego) and just go for it.
I sometimes prefer writing tests after I figured out the top level API.
I sometimes hate myself for the unit tests I wrote.
I sometimes love myself for the unit tests I wrote.
I sometimes omit top level tests.
I write code without any tests!
I will not merge any code without tests into this project!

This is how programmers work, we are all human, we all prefer different approaches. But I think you are wrong @StefanHoutzager

1 Like

I want to work evidence based, not on what one enjoys or not.

Excuse me? Seriously?

“Oh man, that company drained me, 12 hours a day, tight schedules, always asking for more!”
“Ah, I want to work evidence based, not on what one enjoys or not.”

This is what we are dealing with, not some studies.

I gave my own experience, based by me, and I think I made it quite clear that everything is my opinion.

If I can’t be prove for my own experience… wtf can.

Point taken.

So should I tell my employer that we need to hire an expert team to inspect our code before we do the next deployment? Or should I tell them that we it will take longer because we are trying to write reasonable, test based code?

EDIT:

What exactly are you proposing?

This question is very serious. If if you think my own-opinion-based development is wrong, please enlighten me.

And please answer my other replies, even though they were not in the same threat. I can compile it into one if need be.

Dude, if you want to talk to yourself, please just start a blog and get this over with?

It’s the same with you every time:

  1. You express a strong opinion.
  2. You claim it’s supported by science.
  3. You cite a very sketchy paper (which, as I pointed out, doesn’t even qualify for a “paper” in the pure scientific terms).
  4. You are passive-aggressive when criticised:

:point_up: Q.E.D.

  1. You start dancing on the edge of the forum rules by subtly telling people their opinion is bad:

There exists no evidence at all when we’re talking about ways to approach creating and maintaining a software project. There’s no OneAndOnly® solution; it all depends on the project. I don’t know why this is so hard to accept but I am giving up.


Seriously, at this point it speaks bad for me that I am even responding to you. I am taking it as a future improvement remark towards bettering myself.

Again, a friendly advice: if you like talking to yourself and don’t want your opinion discussed, just start a blog and disable comments. :003:

4 Likes

There is a lot of research done concerning evidence-based software engineering and there are outcomes that you can take profit from. For the rest I will neglect your hostile reaction. That means a flag.

Anybody knows how I can once and forever leave this forum? I mean not just logout.

“Evidence-based” is not a stick to beat other people with. Scientific literature is always subject to review: you cannot assume that because something is written in a paper, it’s proved and decided. Papers are part of the discussion, not the end of it. If you are implying that finding problems with a paper means being unscientific, I think you are misguided.

You cited a paper titled “On the Relation Between Unit Testing and Code Quality”. The paper could not find a strong link between unit tests and code quality. I think there are multiple problems with the paper, from the fact that code quality is not an unequivocal physical measurement like temperature, to the small sample size, making generalization at least dubious. Even then, the result is not surprising: I argue that testing is not about code quality, but about keeping cost of change down.

I would not be surprised to find that eating fruit and vegetables does not reduce the risk of getting struck by lightnings, but that does not mean that we should all stop eating them.

Also, beware of your own selection bias when cherry-picking studies.

4 Likes

My mod box is full of flag requests.

Gentlemen, can we please continue this controversial topic without shouting?

It’s ok to disagree, but do it with form :slight_smile:

I believe I haven’t broken the rules but felt I had to call him out. He’s not interested in a discussion and yet, this is a forum.

Do let me know if I crossed the line though, I won’t object.