Why We Test
We test to find bugs, or at least make sure there aren’t any, right? Wrong. Or at least it’s not just about finding bugs. Finding/eliminating bugs is one of the specific things we do as part of testing, but that’s not the goal. It’s not about lines or branches covered. It’s not about having tests pass or dogfooding. All of those are things we should do along the way, but they’re not the reason we test.
I think Dan North said it really well.
The purpose of testing is to increase confidence for stakeholders through evidence.
That’s why the things we do, fallbacks, guardrails, formal analysis, input validations, code reviews, unit tests, code/branch coverage, integration tests, dogfooding, etc., are important. These things give us concrete numbers that describe things we know about the code and the system. They are how we provide the evidence needed.
That data lets us show how we have reduced risk in different ways. They’re not foolproof though. We don’t test all possible combinations or sequences of inputs. Even tools like TLA+ can’t guarantee correctness. All we can do, especially in a complex system is increase our confidence (by reducing risk). Risk analysis is a complex topic itself for another time, but for simplicity, risk and confidence are inversely related. Reducing risk increases confidence.
Just understanding that isn’t enough for the people designing/implementing/running the tests. You have to understand who the stakeholders are and what’s important to them. Stakeholders are everyone impacted by you building the thing under test. Not just the users, direct and indirect, but you as the builder, your team, your manager, and your company. Each of them have different views, different requirements, and different levels of acceptable risk. Depending on who and what you’re doing you could be in many different stakeholder groups. You may also be completely separated from one or more of them. You still need to understand them and what is risky to them.
Because at the end of the day, we’re building things to add value for some group of people. We test to get the confidence that what we’re building will do what it’s supposed to an acceptable level for those people. That’s why we test.
