Test Classification

Whether you are thinking about unit vs integration vs system tests, or build (or save) time vs check in time vs release time tests, what you’re really thinking about is test classification and test hierarchy. Or put another way, you’re thinking about why you’re running that test and what the goal of the test is.

Of course you want the test to pass. And you want that pass to mean something. Even if the result you’re looking for is a failure in the system under test (SUT), you want to see that failure so your test passes. But I’m not talking about what makes a good test. That’s a different topic for a different time.

The topic for today is, instead, what is the purpose of the test. What are level of functionality are you trying to test? Knowing the purpose of the test can help you figure out how to classify it. That can then help you figure out how and when to run it.

First, some basics on test classification. There are many types of tests, but in broad strokes, you can think of them applying at 3 levels, unit, integration, and end-to-end or system. To make things more real, let’s consider a clock application.

Unit tests are tests that validate things at the functional level. The typically live and execute at the function/class level. Do these handful of things work well insolation? Do they do what they say they will do, and handle failure gracefully? They typically take less than a second to set up, run, and tear down. Things at the unit level might include configuration, storage, accessing the host’s notion of date and time.

Integration tests are the tests that validate things at the boundaries of domains, functions, or classes. How does class A work with class B? How does your CRUD layer work with the underlying database? How does your logging/timing/resource management system work with its consumers? These tests might take a couple of seconds, and might require access to some host system, For the clock app, you might test reading and writing configuration with the storage system.

End-to-end or System tests are the tests that validate how the system as a whole works. Does it meet the end user’s expectations, or at least not surprise them greatly when it doesn’t. System tests are the ones that validate that even though a bunch of things failed along the way, the system managed to do the right thing, or at least avoided doing the wrong thing. This is where you’ll test emergent behavior as the different parts of the system interact. It’s often only at the system level that you can test what happens when 4 different things fail in a specific way. Because the system level test is the only one where those 4 different components are working together. These tests can take much longer, and often require the real system, or at least a trusted emulator. For that clock, it might be setting up an alarm and making sure it sounds at the appropriate time.

I’ve mentioned the time it takes for these various test types, but it’s not just time that changes. Cost also changes. Running unit tests is almost free. Just a few CPU cycles and that’s it. System tests, on the other hand, can be very expensive. You need to build the entire system and deploy it. To not just the appropriate hardware, but special hardware that you have extra access to for debugging. That all takes time. Any money. Unless you’re doing manual testing. Which takes even more time and money.

Most tests fit reasonably well into one of these three buckets. If one your tests doesn’t, think about breaking the test up into multiple tests to that it does. Once you know which bucket to put your tests in you can move on to the next step, figuring out when you should be running it. I’ll cover that in a different post.

On the other hand, if most of your tests don’t, think about your test design. If your test design seems reasonable, but your tests themselves don’t fit into those three buckets, think about the underlying system design. If your system is untestable at the unit level, that’s not a testing problem, that’s a design/architecture problem. Fix that first. Then recognize that you’re practicing Test Driven Development.

And that’s a good thing.