What Are You Testing?

When writing tests, whether they’re TDD test, Unit tests, Integration tests, or Canary tests, the first thing you need to be sure of is what you’re testing. If you don’t know what you’re testing then how can you know if your test has succeeded? The next thing to do is make sure that what you’re testing is really the thing you want to test. I’ve seen way to many tests that say they’re testing some business logic, but what they really end up testing is either the storage system or some cache along the way.

Tests typically follow the Arrange, Act, Assert pattern. That means you set things up, you try the thing you want to test, then you assert that the response is what you expect. It sounds simple, and conceptually, it is. It looks something like this:

Arrange

The arrange step is pretty straightforward. Create the thing you want to test. Create any needed resources. Create your inputs. Identify the expected result. It sounds simple, and often is, but there are hidden details you need to be sure of.

Act

The act step is the easiest. Call the method. Capture the result(s). Simple. As long as your code is testable. As long as the method does one thing. One logical thing. The more logical things the method does the harder it is to make it do the act, the whole act, and nothing but the act.

Assert

The assert step is more subtle. Conventional wisdom says one assert per test. At it’s simplest, sure. If you have a method that adds two integers your test should assert that the result you get is equal to the sum you’ve calculated. But if you’re doing some kind of merge of two complex objects then a single assert is probably not enough. But logically it’s asserting that the merged object is what you expect, and that’s one logical assert.

In theory it’s easy to Arrange/Act/Assert. In practice though, it can be hard. For a lot of different reasons. Reasons you need to understand not just when writing the tests, but when writing the code too.

In the arrange step it means setting things up so there’s no variability. Does the thing you’re testing have any dependencies? How do you know they’re working? Are you sure you’re controlling all of the inputs? Not just the function parameters, but also any calls the thing you’re testing might make to get external data. Random numbers. Date/Time, environmental information. All of these things, (and more, need to be controlled. If you’re not you’re probably testing the wrong thing. At best you’re writing a flakey test that will fail at the worst possible moment. Dependency Injection and mocks/fakes are your friends here. Don’t leave the response of a dependency to chance. Making something testable often means adding interfaces and using them so you can replace the implementaiton behind the interface in your tests without changing any other operational characteristics.

In the act step it means arranging things so you can actually call the thing you want to test. It is a method? Is it public? Is it just a branch in some other method and you need to carefully craft the setup to force that branch to be called? If it’s hard to run the thing you want to test then you’ve probably got some refactoring to do. Make the code in the branch a method, then call it. Another common one is separating the code from calling some external thing from the error handling. A common code pattern looks like

retval, err = dbManager.Insert(newUser);
if err != nil then {
    switch err {
        case ALREADY_EXISTS:
           <Specific Error handling Here>
           break;

        case INVALID_OBJECT:
           <Other Specific Error handling Here>
           break;
    }
    return;
}

<Happy Path Code Here>

It can be hard to get an external reference (or it’s mock) to return a specific error just so you can test the error handling. A better choice might be to write it something like

retval, err = dbManager.Insert(newUser);
if err != nil {
    HandleError(err);
} else {
    HandleInsert(retval);
}

That way, HandleError and HandleInsert are both easy to test. Arrange the one input variable, call the method, assert what you want. And the wrapper is trivially inspectable by observation. Add some dependency injection and it’s trivial to test too.

Which brings us to the assert step. There are those that say one assert per test, and that might be an implementation, but the important thing is one logical test per test. And it should match the name. If there are 20 different types of invalid input then there should be 20 tests, each for one type of invalid input. You don’t want one test with 20 different ways to fail. It adds too much cognitive load when you need to figure out why your tests fail. What failed and why should be obvious when you see the name of the failed test.

And again, if the thing you’re testing does so much or has so many side effects that your test name is “validateInputsAndComputeResult” it’s probably time to refactor. In fact, any time your test name has been to Conjunction Junction, it’s probably time to refactor your code to do less.

So when you write your tests (and your code), think about what you’re testing.