Pass or Fail?

We say a test passes when the results are as expected and that it fails when one or more of them is not. When one or more asserts fails. That makes sense. When you take a test in school and get the right answers you pass. If you don’t, you fail.

When you took that test in school and got the right answers who passed? You, or the test? It’s been a long time, but the last time I took a test in school I got enough correct answers. When I got the results, they said I passed. They didn’t say anything about the test itself passing or failing.

So here’s another framing question for you. Why do we say we run tests on our software when as a student we take a test? Why do we say the tests pass or fail for software when we say the student passed or failed? I’ve got some thoughts on why.

I think it goes back to Taylorism, Legibility, and language. We tried to divide by function for efficiency. We tried to make sure we always had good, centralized metrics to tell us we were on track. We execute the tests, not the software to get those metrics.

Taylorism is all about assembly lines, decomposing work into separable tasks, and optimizing repetitive steps for efficiency. That doesn’t make sense for writing code, but it might make sense for testing code. After all, the tests are a separable step, and we repeat testing the code over and over again. So Taylorism leads us to considering the tests to be their own thing, completely separate from the code.

Legibility is the idea that the only way to make sure you do things efficiently is to centrally plan, record, and respond to deviations from the plan. To do that you need have the results of all of the phases. You break the coding into tasks and give them a state. You identify all of the tests and give them a state. Since the tests exist for a long time and, according to the plan, never change, the state of the test is the result of the last time the software was tested, pass, or fail.

And then there’s language. In this case, it’s the language of executables. Tests are code. You run code. Testing code, especially unit tests, but even integration and beta tests, are code you run. The code doesn’t run the tests, the tests run the code. The Sapir-Worf Hypothesis says that language influences thought. The language makes us say and think about running the tests to get results instead of testing the software and seeing if the software passes or fails the test(s).

That’s how our frame gets us to talk about running tests and tests passing of failing. Its interesting, and mostly harmless, but its not completely harmless. It’s not harmless because the frame does more than just define the words we use. It also influences how we think of of our software and it’s state. We say the test failed. We don’t say the software failed. That frame makes us think of fixing the test to make it pass, not fix the software. Sure, we usually fix the software, but since the test is what’s failing it makes complete sense to change the test until it passes. The frame makes us think that might be the right thing to do. But it’s probably not.

So next time you find yourself talking about tests failing, consider reframing it and talk about the software failing the test. Because how you look at things influences what you do about it.