Hofstadter's Law

"Hofstadter's Law: It always takes longer than you expect, even when you take into account Hofstadter's Law"

Let's be honest. Estimation is hard. It doesn't matter if you're talking about digging a ditch, building a house, doing year end perf feedback, or planning a sprint, let alone a whole software project. We estimate and we get it wrong, over and over again. There are lots of good reasons for it. From rocks, roots, and pipes where you want the ditch to shifting software requirements, you don't know what you don't know until you know it. Then there are the psychological aspects.

The Planning Fallacy, which says we don't consider past examples when we make estimates and that we ignore hidden complexities, can lead us to underestimation. Even when we take that into account, there's Optimism Bias, which says the future will be better than the past. So again, we under-estimate.

So what's a poor software developer to do? First and foremost, be aware of your own biases. Second, make sure you understand what you're estimating. How much like something else is it really? How significant are the differences? What are the external influences that are going to bend the design halfway through? Third, use your history. If you (and your team) are always 25% under, don't change anything (NB: This is hard. You'll want to reduce the estimate so the adjusted number looks right. Resist), just add 25%. Fourth, break things down. If you can't break it down into smaller tasks you don't understand it. If you don't understand it the unknown unknowns will kill you.

Finally, avoid feature creep, or at least acknowledge it and add some time to the schedule for every addition/change/update to what you're doing. Sure, that new idea is important, and you should probably do it, but it will take time, so account for it.