More Error Types

I’ve talked about Type I (False Positive) and Type II (False Negative) errors before. While it would have been so much better if they just called them False Positive and False Negative cases, they only cover part of the problem. A more complete list would include the Type III (the right answer to the wrong problem) and Type IV (the right answer for the wrong reason) errors.

The Type III error ought to be innocuous. After all, you may have wasted some time getting to an answer, but you get to a correct answer. Using that answer is going to be good, right? As usual, It Depends.

To use a car analogy, you’re driving down the road and you hear a thumping noise. Nothing seems obvious, so you keep going. A few miles later, the engine dies, and you safely get the car to the side of the road. After taking a few minutes to calm down, you get out of the car and walk towards the front of it. You notice that the hood isn’t fully latched down. So you open the hood, slam it properly, and confidently get back in the car to continue down the road. Unfortunately, when you go to start the engine, it won’t start. Yes, the hood wasn’t latched, and it may have been causing the noise, but fixing it didn’t help get the car running at all.

Since that didn’t work, you look around some more. You notice that there’s a loose spark plug wire. That’s surely a problem. You fix it, but the car still doesn’t start. You keep finding and fixing things, but you can’t get the engine to start. It turns out that that answer to the question of why the engine died is that you ran out of gas. All those other things, which are related to the car’s operation, don’t help. They are problems, and you came up w/ solutions to them. They answered the question of why there was a thumping noise. They answered the question of why the car was running roughly. But they didn’t answer the question of why the engine stopped running. You’re still sitting on the side of the road.

So before you go accepting whatever answer you got, make sure it’s the answer to the question you asked. Otherwise, it’s not going to solve your problem. And, might make things worse.

The Type IV error can be even harder to spot, and can do even more damage later. But why? You got the right answer, and it solved your problem. Yes, it did, but it also taught you something that isn’t true. It’s now one of the things you know that just ain’t so. And that’s one of the hardest biases to fight.

Going back to the car analogy, one day your car is pulling slightly to the right. You remember from watching NASCAR races that you can change the handling of your car by changing tire pressure, so you check, and the right front tire was a bit low. You add some more air to the tire. The problem goes away. You’re happy. Over the next few months, it happens a few more times because there’s a small leak in the tire. When the car starts pulling you know it’s time to add some air. When you get new tires there’s no leak, and you don’t have any more pulling. Then one day the car starts pulling to the left. You use your new knowledge and add air to the left front tire. The problem goes away, and you’re happy. It doesn’t come back and you forget about it. Months later you find that the left front tire is completely worn out in the center because it’s been overinflated. The car wasn’t pulling to the left because of a low tire, it was because of that pothole you hit, which slightly bent the tie rod. So now you have two problems. A bald tire and a bent tie rod.

If it hadn’t been for that Type IV error, the right answer for the wrong reason, you wouldn’t need to get at least 2 new tires and and a tie rod. If you hadn’t known that inflating the tire would solve your pulling problem, the right answer for the wrong reason, you probably would have dug a bit deeper and solved the real problem.

So before you go and use the solution you already know works, make sure it’s the solution for the current problem, not the solution for another problem with the same symptoms.