Why is this hard?

Thanks for sharing. I’ve added your site to my RSS reader

Personally I’m not really convinced estimates in software are possible at all. I think we might need to simply take everyone along for the ride so they can see first-hand how the design is unfolding.

Same, and that’s proven by mine and many others’ professional practices. I had calls, on multiple occasions in the past, with colleagues about why weren’t some 2-3 problematic items inside a PR surfaced earlier during a tech planning stage. Mine, and many others’, answers were stereotypical: “Because even if we sat on a call for two hours and discussed a task breakdown, we still wouldn’t guess X and Y will become a problem. The only way for this to surface would be if we started coding it together during a screen share session, and even that does not guarantee we’d have spotted it earlier than we have in this PR”.

I get the need to want to reduce chaos but this is similar to trying to observe molecules and atoms – the deeper you go, the blurrier it gets. You have to stop while the clarity is at a satisfactory (== not ideal) level and not pursue further because you’ll hit the “last 20% of the task take 80% of the energy” part of the Pareto principle head-first, and it’s going to hurt.

I once worked in an environment where estimates worked. There were various factors involved that I would say contributed to this, some which many people would dismiss or just not want to work that way. Heavy sandbagging was also a factor. But they were successful in that my team has a solid understanding of what we were able to get done, communicate that to leadership, and we’d always hit our goals or deliver early and everyone was happy. So long as the org understands that estimates are, in fact, just estimates and the goal isn’t speed but meeting expectations, they actually can work very well. But again, I’ve only had this experience in once.

I think, and please correct me if I am wrong, that you are both saying that perfect estimates are impossible. I agree.

However, are estimates supposed to be perfect? My dictionary is using words like “roughly calculate” and “an approximate” in the definitions. I don’t think perfect is the goal.

I believe estimates are a valuable tool in what we do. I can’t imagine trying to run a business without knowing roughly how much money, time, or employees our work was going to take.

I also believe we can find ways to get better at estimating, even though it is really challenging. We’ve built mostly reliable computer networks for machines we can’t even trust to correctly tell us what time it is. Just one of the techniques we use to get around that is to be explicit about where the error bars are. “I think it’s this time but I could be off by as much as…” Could we not try to do things like that for our estimates?

…this of course completely redefines what the word “estimate” means, which I think is the real problem with them, because even when people say that understand what the “estimate” means, I they still think it means “the time it will take give or take an hour or two.” I catch myself in this thinking as well.

My experience is that most people ask for estimates because they are trying to line up when work can be started on the next feature or phase of the project, or when they can announce a release date. But that is not how software works. We’re not building bridges, we’re building dynamic ever-changing systems. Even an experienced developer who may have been working on the project since day one doesn’t have the information required to come up with a number that will be suitable for planning when the job will be finished. The best we can do is try something and see what we get, then reassess.

I’m thinking that instead people need to have the concept of investing in a software project in stages. We’re exploring it together. It might turn out that we decide to stop the project and you have to write off the costs so far given the information learned, or adjust the priority or scope of this or other features. Sort of like hiring someone to come up with a blueprint, except the blueprint is also a functioning system. The system can be refined/expanded from there. That’s better than pretending the job will be complete in a certain time period but then running into all the problems when it is not.

Problem with many business people is that they don’t hear “he estimated the deadline”. They hear “he committed to the deadline”.

This is one of points I’ve experienced some people to be dismissive of. Really your whole second paragraph. For me there was a lot of prototyping, and actually throwing the pototype away. Not “wink-wink-nudge-nudge we’ll just re-work it a bit”, but throwing the whole thing away, building estimates on what we learned (this is after watching customers play with the prototypes), and starting completely from scratch. From there (and this still isn’t the whole picture) it’s not as difficult to build decent estimates.

Pro tip: if you call every variable, function, module, class, etc foo, foo2, foo3 and so on, and (if you are a frequent committer) every commit message is WIP, it makes it much easier to let go of prototypes

Ya, leadership has to understand this or it doesn’t work. But also, that’s why sandbagging is key

You need to know your audience. If your peer ask for an estimate, you give the mean. If your manager asks for an estimate, you give mean plus 1 sigma. If your customer/sales asks, you give 3 sigma out.

I think one reason so much time is spent wrestling with these questions even though at this point we all have a pretty good grasp on the situation is it’s hard for technical people to admit that not everything is a technical problem (or more fairly perhaps, keep it constantly in mind). Estimates are not hard because software is “complex.” Building bridges is a complex process as well with plenty of unknowns. We are all experts at breaking down complex problems. It is hard because of power dynamics. Managers, clients, etc don’t “misunderstand” estimates because they are non technical but because they have a different set of vested interests. This is why most effective strategies for dealing with these problems are social/psychological and not technical. Does anyone here really doubt the core ethos of agile? Or is it rather that any technical process can just be turned into a tool of manipulation/control in toxic organizations, i.e. organizations with unstable (unjust?) power dynamics? In my experience it is quite clearly the latter.