Code Coverage Is Not Enough

Tales from the real word today. Code coverage is important. It can give you confidence. It can help you move faster. It can alert you to problems before they get exposed to your users. But it’s no guarantee of correctness.

Bad Tests

The first problem is that 100% code coverage does not in any way guarantee that the tests you’re using to measure your coverage are valid tests. Writing good unit tests is hard. A unit test that calls a function then asserts true and returns gives the same coverage as a well written unit test. There’s enough to say about the writing of good tests to fill up at least 3 more posts, so I’ll leave that for another day.

Transitive Coverage

Unit tests almost always have some transitive dependencies. Unless you’re careful, you’ll end up counting the coverage of those transitive dependencies as part of your code coverage. Because you aren’t trying to test those dependencies, only how you act with them, you get very little signal on the correctness of them. But since you’re just looking at “was this line executed?”, you get coverage. You don’t know if that coverage comes from direct testing, where the coverage means something, or from transitive coverage where it means nothing. That erodes a bit of confidence.

Branch Coverage

Another problem you’re going to have is complex branching statements and short circuiting logic. It’s part of writing good tests, but if you’re and’ing 4 booleans together there are 16 ways you need to test it. Add in some or’s and groupings, and it’s very easy to test both sides of a branch, but not really validate that you’ve covered all the possible cases. Another bit of confidence gone.

Integration Coverage and Emergent Behavior

But even if your unit tests are good, and you have 100% coverage of the source lines, you don’t necessarily have coverage of all the possible combinations of those things. As your system gets bigger, and the components interact in different ways, you get more and more emergent behavior. And your unit tests don’t validate ANY of those interactions. So you lose another bit of confidence,

But, But, But

If code coverage doesn’t guarantee correctness, why should you care? Why do ISO-26262 and MISRA talk about it? Why should you bother to measure or do anything about code coverage?

Because while code coverage doesn’t guarantee correctness, changes in code coverage very reliably tells you when something has changed. It helps guide you to where there is more work to be done. When you follow that guidance, you can improve your confidence in the correctness of your code. And that’s what makes code coverage valuable.

And one last thought on testing-based coverage. Don’t forget, testing can’t prove things are right. It just proves things aren’t wrong as your tests define wrong.