Optimizing Tests

Like most things engineering, the answer to the question “Is this a good test?” is it depends. Tests have properties and the different kinds of tests make different tradeoffs between those properties.

While there are a lot of different properties a test could have, some of the most important are:

Deterministic: A good test gives the same result every time. Which usually means it has no external dependencies.

Fast:Tests should be fast. How fast? It depends. Programmer/Unit tests should be single digit seconds. Integration/acceptance tests should be single digit minutes. Bake/Soak tests might take hours or days. And of course this is for individual tests. The sum total of all tests of a given type, even with parallelization, might be longer.

Independent: Your tests shouldn’t have side effects. You should be able run them in any order and get the same results. Adding or removing a test shouldn’t change any other results. This means that you need a good way to set up all the preconditions for a test.

Specific: If a test fails, knowing which test failed and how should be enough to isolate the problem to a handful of methods. If your test that includes generating a value, storing it, and retrieving it fails, you don’t know which part failed and you have to examine the entire system to understand why. Much better to have tests for each part so you know where the problem is when the test fails.

Two-Sided: Of course you want to test that valid inputs give the correct results. But you also want to test that invalid inputs give the expected failure.

Uncoupled: Tests shouldn’t be concerned with the implementation of the solution. Ideally you would mock out an entire system and have it be functional and inspectable. We’ve done that for our in-memory file system we use for testing things on the infra team. We can preload the system, read/write arbitrary things, and then see what happened. On the other hand, for some things, like network calls, our mocking system looks for certain patterns and responds in certain ways. Not ideal, but a compromise. And avoid a mocking system that just returns a canned set of responses in a specific order. That’s both brittle and not representative of the thing you’re mocking.

Finally, going back to the classes of tests, and the different tradeoffs. Unit tests are run frequently. You might trade off testing how things work together for speed of testing. On the other hand, an integration test might have a more involved setup so you can test the interaction between components.

So what’s important to you when writing tests?