Stop designing up front the moment you start speculating.
– Kent Beck
That’s a pretty bold statement. After all, how much do you really know about what you’re going to need to do at the beginning? At the beginning of a project (or feature, or task, or bug investigation) all you know is a rough idea of the problem to solve, not how you’re going to solve it.
To me, what Kent is saying isn’t that you should only do design on things that are known/fixed/unchangeable, but that the level of detail of the design should match the level of detail of what you know.
As an example, if you’re in the US, and you need to be in England in 2 weeks for 3 days, making the decision to fly instead of going by ship matches the level of understanding. Adding detail to that decision, i.e., deciding what day/time to fly on, or which airport you’re going to fly into, requires more information that you have. That info is probably available, and easy to get, but deciding to leave at 8 AM on Monday morning before you know if you need to be ready to work Monday morning in England. Similarly, deciding to fly into Heathrow before you know that you need to be in Edinburgh is just going to cause you to have to change your tickets later.
It’s the same with software. You have the problem you want to solve. You have some understanding of the situation as it currently exists. You cad (and should) do a design to the same level of detail. A handful of boxes on a whiteboard and some lines and arrows connecting them. You might have persistent storage. You’ll probably have a few different major steps in the process of solving the problem. There’s an expected flow between the steps, and some places where you might need to back up or start again. If you’re like me, you’ve already thought of 15 different ways things can go wrong, so you know you’ll need a way to get information about a failure late in the process back (or forward) through the system. Given what you know, that’s about how much you can design. After that, you’re speculating. You’re guessing really. It might be an educated guess based on past experience, but it’s still a guess.
That’s the point to stop designing and start implementing. Build some APIs. Let some test data flow through the system. Hardcode some good and bad responses and see how the system responds. Adjust your APIs and data structures so things flow well in both happy and sad cases. Learn from the system you’re build. If something seems hard to do with your flow, it’s probably not you. It’s the system telling you that you’ve uncovered an emergent property, and you need adjust to work with it. Once you’ve got it working, look again. Do some refactoring and cleanup. Listen to what the system is telling you and adapt. It’s Test-Driven-Development at work. Or if you don’t believe in that, you can call it Domain Driven Design. Or Behavior Driven Development. Or even Extreme Programming.
It doesn’t matter what you call it. The important part is that you design to what you know, leaving room to adjust as you learn more. Then you learn a bit, design a bit, and do it all again. Rinse, Lather, Repeat. It’s more than a way to sell more shampoo.
Don’t worry (yet) if it can handle 10K transactions/sec or if it’s resilient to a backhoe taking out a datacenter. You might need that. You might even know you’ll need it for a production system, and you might be right, but you don’t need it yet. So don’t worry about it. Don’t preclude it, but don’t code yourself into a corner where you are stuck with a decision you made before you knew what tradeoffs you were making.
Beware of biases. There are lots of them. Sunk cost bias. You’ve put so much effort into making this thing work you don’t want to give it up. Recency bias. I just finished this, so I need to use it now. Overconfidence Bias. I know what I’m doing, and I know everything I need to know already, so my initial idea must be the right one.