Recent Posts (page 1 / 67)

Simple or Easy?

Here’s a question for you. What’s the difference between simple and easy? Are they different, are they the same, or is one a superset of the other? If you had to choose one, which would you choose? And why?

First, let’s see what Sir Merriam-Webster’s reliable book has to say.

SIMPLE

readily understood or performed
    simple directions
    the adjustment was simple to make

EASY

requiring or indicating little effort, thought, or reflection
    easy clichés

Similar, but not quite the same. It’s kind of like the difference between complex and complicated. Complicated processes are often made of many easy steps to do. You just need to do them correctly and in the correct order.

Complex things, on the other hand, are often simple. A Foucault Pendulum is simple. It’s just a weight on the end of a string. Describing its motion, on the other hand, is very complex. It’s a function of the weight, the length of the string, and the Earth itself, including the pendulum’s position on the Earth. Once you know about it, it’s simple, but figuring it out from just watching the pendulum swing is very hard.

It’s often like that in software. Complicated systems, tools, and pipelines are tedious to build, and hard to get right, because they’re so exacting. Bash scripts are very often complicated, but because they’re scripts, each step is easy. Distributed systems are simple. Just do things in parallel on different machines. Do the exact same things you would do on one machine, just do half of them on one machine and half on the other. Simple. But complex because the failure modes are multitude, and the differences between them are subtle.

To answer the questions:

Are simple and easy the same thing?

No, they’re not. Simple is easy to understand. Easy takes little effort to do. But beware, not everyone will agree of something is simple or easy. Or why.

Is one a superset of the other?

No. They’re related, and something can be both simple and easy. In fact, many things that are simple are easy, and many things that are easy are simple, but being simple does not imply being easy, nor does being easy imply being simple. There’s not a causal relationship between them.

If you had to choose one, which would you choose?

Of course, It Depends.

When I’m starting out on something I’m unfamiliar with, I want to make it easy. There might be lots of steps, and it might be complicated, but until I understand the system, I want it to be easy. Every part should do one thing and one thing only. Interactions between parts should be minimized. If something is not right, I want there to be only one place to look for the problem, and I want to be able to work on that one thing and not have to worry about breaking anything else. Of course, doing things often ends up with a brittle system that is easy to use, if you use it just right.

Later, for the same project, when I’m more familiar with the domain, I’m going to want to make it simple to use. The internals will become more complex, with interactions that require deeper understanding to deal with problems. It won’t be as easy to understand the intricacies, but the surface will be simple. Using it will be robust and easy to understand. It won’t every surprise you, even if you misuse it, and will be hard to misuse.

But what if you REALLY had to choose just one?

In that case, I’d choose easy. Because easy takes little effort right now. So I can add value. And if I can keep it easy, I’ll always be able to add value. And when you do that, you usually find that keeping it easy eventually makes it simple.

Governing the Commons

Adding to my book reviews, consider Governing the Commons by Elinor Ostrom. You might wonder what a book about Turkish fisheries, Swiss grazing pastures, Japanese forests, and Spanish and Philippine water systems has to do with software development. It does seem to be a bit of a stretch.

Some background first. The common part is all about what the Commons actually are. In this case, it’s the same commons talked about in The Tragedy of the Commons. A shared resource. Traditionally, that’s used to denote a shared, finite, natural resource. Like a pasture, a forest, or water. That’s the commons or, as Ostrom says, the Common Public Resource (CPR).

In software development, particularly large projects with multiple teams, but really, any project with multiple developers, there are many shared finite resources. The most obvious are time, IO bandwidth, and storage (RAM and long term). You can probably list others, but as you can see, software has its own CPRs.

Those tangible things aren’t the only CPRs though. There are also more intangible things. Things that make up Internal Software Quality (ISQ). Like architecture, naming conventions, coding styles, and domains and boundaries. Those things may not be finite, but they are on a common, public, and they are on a continuum that is influenced by local actions with regard to the whole.

That’s how software development is like the Commons, and Ostrom’s findings and conclusions can tell us something about how we might do things differently to encourage better results.

Conventional wisdom tells us that when you have a group of actors, each looking out for their best interests, who must share a finite resource, they quickly use it all up, because each actor is working toward their own local maximum. The each want to get as much of the resource as they can, and before anyone else can get it. Or, one actor ends up controlling the resource, to the detriment of everyone else. That’s the tragedy. And the most common “solution” to the problem, is heavy handed, external (often governmental) regulation. It sort of works, but leaves everyone looking for a way to game the system and get the most for themselves, even if it makes things overall worse for everyone.

What Ostrom found, after looking at the various successfully managed CPRs, was a list of 8 guiding principles:

Group boundaries are clearly defined

While the shared owners and individual teams may be changing, everyone agrees on who is in the overall group, and which sub-group they’re in. It’s self-organized and dynamic, and often based on unwritten tribal knowledge, but the organization can be clearly seen.

Rules of use are matched to local conditions

Since the resources under consideration are different, the rules around their use are specific to those resources. It’s not as simple as saying “There are 1000 users. You each get 0.1%.”

Most (all?) actors are involved in modifying the rules

If you’re in the community, you’re automatically part of governing the community and the resource. If you’re not in the community, you’re kept out of the resource, until you join. Joining may be complicated, but it doable.

The community is allowed to set and enforce the rules

The converse of the above. If you’re not in the community, you have little (or no) say. External influences (like a government) are influences only, and only have influence on group members, not the group itself.

The community monitors itself

The community is tracks itself and identifies and “grades” infractions. Again, this is often known but unwritten, bit for larger communities it may be written and public.

There are sanctions for breaking the rules

If you break the rules, there are consequences. The bigger the offense, the bigger the consequence. Up to and including being excluded from the community

The community manages conflict resolution

The community handles its own disputes. You can bring up a dispute you’re involved in, and the community can step in where there are unresolved disputes, but either way, it stays internal

It’s a system, so communities and CPRs can be nested

The only way to scale to large numbers of actors or groups of actors is to have a hierarchy and nest groups and portions of the commons inside larger groups. The challenge here is to maintain the other 7 principles while building and operating the hierarchy.

Sure, when Ostrom wrote this she was talking about natural resources, but software development is a socio-technical endeavor. As such, there are lots of CPRs that need to be managed. Applying these principles can help us maintain those resources at an appropriate level while ensuring that everyone, both individually and collectively, have the right amount of those resources.

Finally, there’s a download of the book available on archive.org. The formatting is pretty bad, but the words seem to be correct.

More Error Types

I’ve talked about Type I (False Positive) and Type II (False Negative) errors before. While it would have been so much better if they just called them False Positive and False Negative cases, they only cover part of the problem. A more complete list would include the Type III (the right answer to the wrong problem) and Type IV (the right answer for the wrong reason) errors.

The Type III error ought to be innocuous. After all, you may have wasted some time getting to an answer, but you get to a correct answer. Using that answer is going to be good, right? As usual, It Depends.

To use a car analogy, you’re driving down the road and you hear a thumping noise. Nothing seems obvious, so you keep going. A few miles later, the engine dies, and you safely get the car to the side of the road. After taking a few minutes to calm down, you get out of the car and walk towards the front of it. You notice that the hood isn’t fully latched down. So you open the hood, slam it properly, and confidently get back in the car to continue down the road. Unfortunately, when you go to start the engine, it won’t start. Yes, the hood wasn’t latched, and it may have been causing the noise, but fixing it didn’t help get the car running at all.

Since that didn’t work, you look around some more. You notice that there’s a loose spark plug wire. That’s surely a problem. You fix it, but the car still doesn’t start. You keep finding and fixing things, but you can’t get the engine to start. It turns out that that answer to the question of why the engine died is that you ran out of gas. All those other things, which are related to the car’s operation, don’t help. They are problems, and you came up w/ solutions to them. They answered the question of why there was a thumping noise. They answered the question of why the car was running roughly. But they didn’t answer the question of why the engine stopped running. You’re still sitting on the side of the road.

So before you go accepting whatever answer you got, make sure it’s the answer to the question you asked. Otherwise, it’s not going to solve your problem. And, might make things worse.

The Type IV error can be even harder to spot, and can do even more damage later. But why? You got the right answer, and it solved your problem. Yes, it did, but it also taught you something that isn’t true. It’s now one of the things you know that just ain’t so. And that’s one of the hardest biases to fight.

Going back to the car analogy, one day your car is pulling slightly to the right. You remember from watching NASCAR races that you can change the handling of your car by changing tire pressure, so you check, and the right front tire was a bit low. You add some more air to the tire. The problem goes away. You’re happy. Over the next few months, it happens a few more times because there’s a small leak in the tire. When the car starts pulling you know it’s time to add some air. When you get new tires there’s no leak, and you don’t have any more pulling. Then one day the car starts pulling to the left. You use your new knowledge and add air to the left front tire. The problem goes away, and you’re happy. It doesn’t come back and you forget about it. Months later you find that the left front tire is completely worn out in the center because it’s been overinflated. The car wasn’t pulling to the left because of a low tire, it was because of that pothole you hit, which slightly bent the tie rod. So now you have two problems. A bald tire and a bent tie rod.

If it hadn’t been for that Type IV error, the right answer for the wrong reason, you wouldn’t need to get at least 2 new tires and and a tie rod. If you hadn’t known that inflating the tire would solve your pulling problem, the right answer for the wrong reason, you probably would have dug a bit deeper and solved the real problem.

So before you go and use the solution you already know works, make sure it’s the solution for the current problem, not the solution for another problem with the same symptoms.

Strong Opinions, Loosely Held - Part 2

A while ago I talked about Strong Opinions, Loosely Held. I talked about the problem with people with structural power (the HiPPO) having strong opinions and how those opinions often get too much weight.

As Jim Barksdale of Netscape once said,

“If we have data, let’s look at data. If all we have are opinions, let’s go with mine.”

Doing that can easily get you into trouble, because being a HiPPO doesn’t make it right. In that post I only teased about the second part, so it’s time to talk about that.

After making sure other opinions get sufficient space to be investigated and thought about, the most important thing you can do with your strong opinions is to hold on loosely, but don’t let go to soon. You have that opinion for a reason. It might be a good one. It’s probably based on some experience you had or data you’ve seen.

But that doesn’t mean that it’s still the right opinion. You need to make sure that it’s still the right opinion. You need to check your biases.

Like confirmation bias, where you only look at data that supports your opinion. You need to look at that data, but you also need to look at contrary data. Especially when it’s brought by someone with a different opinion.

Or recency bias. Something like this just happened, so obviously it’s going to be like that next time, right. Possibly, but not definitely. Maybe what just happened was a black swan event. Or the circumstances were different.

Optimism bias is another one. Sure, it didn’t work last time, but we’ve learned a lot and we can be better/faster/stronger than last time. We can make it work now. Maybe, but maybe not. You need to be honest with yourself about your capabilities.

Ego bias is a big one. Especially for HiPPOs. After all, if they’re the highest paid, they must be the smartest/most valuable/most correct, so their opinion must be the best. That might be true for some set of conditions/areas, but no single person is the smartest in the room at everything. Again, honesty is the antidote to bias.

There are many other biases and reasons why your opinion might be wrong. The trick is to not be so tied to your opinions that you’re not able to listen to reason. To listen to another opinion.

Indecision can cripple a project/team. But so can slavish devotion to an opinion that has been proven to be incorrect. So have a strong opinion. Present it with confidence. But don’t force it on others. Encourage a discussion. And when you learn new facts of and you see that your opinion is not helping you, allow yourself to change your opinion.

Then work from that opinion/decision. Until there’s a good reason to change it.

Virtuous tests

I’ve talked about the different kinds of tests. I’ve talked about when to run the different kinds of tests. I’ve said that your tests need to be good tests. All of that is true. What I haven’t done is talk about what makes a good test.

Of course, the answer to the question of what makes a test good is, It Depends. Mostly it depends on what kind of test you’re writing and when you’re running it. System tests are different from integration tests, which are different from unit tests. They do, however, have a few things in common. While your code should be SOLID (or CUPID), your tests should be FIRST. Your tests should follow the Arrange, Act, Assert pattern.

Those are all structural guidelines though. What helps separate great tests from just good tests? Tests are code, so they should have those virtues. They should also have a few more that only apply to tests.

Have a good name

Regardless of what you’re testing, your test should start with a good name. You want to be able to look a list of failed tests and understand exactly what failed. The name of the test needs to tell you what’s being tested. If you’re tests are called TestA, TestB, and TestC, you might know what a failure of TestC means 10 minutes after you wrote the test, but when if fails in 3 moths due to a bad refactor you (or whomever is running the test), will have no idea what failed. Even more specific, a good test name describes what is being tested, and tells you about the test case and the expected result.

Test one thing at a time

This makes the first thing easier. A test should only validate one thing at a time. Through good use of fakes, mocks, and dummy components, you ensure that the result of the test only reflects what happens with the thing you care about. Everything other than the thing under test should not be able to fail. (Yes, I know there are cases where this is impossible. Especially when there are noisy neighbors, but do the best you can).

Control the environment

You need to control the environment. Things like time and date, random number generators, environment variable, and disk and memory status. Also, you can’t rely on timing (unless the purpose of the test is to make sure things take less than a specified amount of time). Depending on the language and test framework, you might need to have a shim around your date/time/random library so you can control things. For single threaded code, you can rely on ordering, but for any multithreaded code, good tests handle the asynchronous nature of threads. And make sure they’re all done/cleaned up.

Validate the response

This should probably go without saying, but needs to be said anyway, the results/side effects should be checked. In almost all cases, just checking that no error was thrown/returned isn’t actually checking anything and isn’t a good test. Whatever you do, check something. assert (TRUE) unless there’s a panic/crash is almost always an invalid test.

Group different variations into a larger test

Tests should be grouped together based on what they’re testing. A good rule of thumb (for unit tests) is to pair a test file with each source file. It keeps things close together when you need to work on both of them (write the test, the write/update the code that makes it pass). But not just where the tests live, but how they’re grouped.

For unit tests for example, you want to test not only the happy path for a method, but also the various unhappy and edge condition paths. You could write a special test for each one, but that’s likely to end up with lots of duplicated code that’s hard to maintain. A better choice would be a table-driven test. Various languages/frameworks have tools to help, so use the one that makes sense in your case, but in general, instead of multiple tests, a single test that calls another method with a set of parameters that does the actual test. Then, when you discover a new test case, you just update the table with the parameters and expected result (or error). This also helps with the first case. You can usually concatenate the base function’s name with the test case’s name to define the test you’re running. That makes it very easy to understand what went wrong and where to look for the code/data that’s failing and the code being tested.

For integration or system tests, this might end up being a set of scenario definitions that are called as a single system test, then each scenario gets its own result. Again, the base name tells you what overall thing you’re testing, and the scenario name tells you the specifics.

Clean up after yourself

Since your tests could run in any order, any number of times, you need to make sure the results of the last run don’t pollute the results of a new run. The most common way this happens is when tests leave things laying around on disk, but extra rows in a database are almost as common. So spin up a new storage system (disk folder, in-memory file system, database, etc) for each test, or make sure that you clean up the shared one, regardless of how the test ends.

Not just the data, but connections, file descriptors, ports, memory, anything your test grabs on to needs to be put back. Otherwise, sometime, when you least expect it, your tests will start to fail because some resource is no longer available. And that will happen at the least favorable time, occasionally. Making it very hard to reproduce and debug.

So next time you’re writing some tests, whether TAD or TDD, make sure your tests are not only correct, but also virtuous.

Code Virtues

I’ve talked about code smells before. Just the other day I talked about comments as a smell. What about the opposite? Are there code virtues?

Of course there are. Things that we want our code to have. In fact, the Pragmatic Programmers published a list back in 2011

Because English is a slippery language and, as Humpty Dumpty said, words can mean what we choose them to mean, each virtue is given as not only what it is, but what it isn’t. It gives you a continuum so you can tell what to avoid as well as what to move toward.

There are lots of details in the article, but touching briefly on each of the virtues:

Working: as opposed to incomplete

Straight from the Agile Manifesto, code needs to work. If it ain’t done, it isn’t adding value, and doesn’t have much virtue.

Unique: as opposed to duplicated

Is it DRY (Don’t Repeat Yourself)? Only have one way to do something. If you need to change something, have it in one place. This goes beyond constants and functions. Keep your domains clean and respect their boundaries.

Simple: as opposed to complicated

Be straightforward. Following on from Unique, have decisions made in one place. Have functions do one thing. Process collections instead of multiple individual items.

Clear: as opposed to puzzling

This one is a bit harder. I would have said Clear, as opposed to clever. Avoid neat tricks. Don’t surprise the reader with side effects. As John Scalzi once said, the failure mode of clever is a**hole.

Easy: as opposed to difficult

Don’t make things hard on yourself. For example, when you go from doing one thing to doing two, it makes sense to just add an if/else. When you add the third, that might still make sense. After that, some kind of table or data driven approach makes more sense. Another thing that makes things easy is automation. Once you figure out how to do something, automate it. It’s easier, and it’s harder to get it wrong.

Developed: as opposed to primitive

One of the big things we do when programming is deal with complexity. There’s some level of complexity you can’t get out of the system. What you can do is make sure the complexity is in the right place and covered with the appropriate abstractions. If your domains are set up correctly, you can use the domain and never need to deal with their internal complexity.

Brief: as opposed to chatty

Be succinct. For whatever succinct means in the language you’re using. Be idiomatic. Those conventions are shared for a reason. It makes your code easier to read and understand.

Now go forth and write virtuous code.

Motivation

Motivation is a funny thing. If there’s a lot of motivation, it can make us do certain things. It can make us not do certain things. It can make us try to do everything. It can make us do nothing. It can make us focus solely on one thing, It can blind us.

On the other hand, if there’s no motivation, nothing happens.

Motivation is funny in another way. There are two kinds of motivation. There is internal and external motivation. Doing something for what you give yourself (internal motivation) and doing something for what someone else gives you (external motivation).

Doing something for what you give yourself just feels good. It’s interesting. You learn something. You enjoy it. You have Autonomy. It can lead to Mastery. And those are it’s Purpose. That’s serious motivation. That’s Drive

You can also get that from doing something for what others give you. You could have Autonomy if the other person sets the target, but you pick the path. You can have Mastery by learning something along the path you pick. And you can take on (or agree with) the other person’s purpose. You just have to pick the right person and the right target/purpose.

What extrinsic motivation can give you, that intrinsic motivation doesn’t necessarily have, is guardrails. Intrinsic motivation can (not always, but sometimes) push you towards local maximums. Without sufficient self-awareness, intrinsic motivation can lead to nerd sniping. You choose the most interesting path. You learn the most. You fulfill the specific purpose. But you missed the bigger goal you started working on.

Sure, you can (and should) keep track of the bigger goals and be aware of when you’re nerd-sniping yourself, but that’s hard. External motivation provides that awareness for you. Or, at least gives you an external point to check against. And because it’s external, there’s something you don’t get if you don’t meet the goal. It could be anything from a badge on a website to an atta-boy to a paycheck. Whatever it is, if you don’t meet the goal, you don’t get it.

That’s the key difference. That’s the guardrail. You don’t get something. With intrinsic motivation you can easily end up changing the goal/purpose so that you are getting your immediate needs met. You climb that hill right to the local maximum, and you’re stuck. No matter which way you go from there, it feels worse, because every direction is downhill.

With extrinsic motivation you can’t do that. That local maximum becomes a bias in a direction, and you might go part of the way there. Eventually, though, the bias is overcome by the pull (or push, depending on how you view things) towards that external goal gets you back on track. It helps keep you from getting stuck on along the way.

There’s also another thing to keep in mind. While your capacity to do things is limited, your motivation doesn’t have the same limits. And intrinsic and intrinsic and extrinsic motivation come from two different suppliers. They’re not a Zero-Sum game. You don’t have to give up on one to get the other. More is better than less, and you want to balance them.

When they’re balanced, you not only meet the external goals and get those rewards, you also meet your internal goals, and get those rewards.

And that can help reduce or even avoid burnout. Which is good for all of us.

Comments Are A Smell

I’ve talked about comments in the past. Comments are a code smell, right? Maybe, or maybe not. It depends. Like so many things in the software world, a recommendation phrased as a comparison has been turned into a binary flag.

Why that happens is understandable. Binary is easy. True/False. 1/0. So easy even a computer can understand it. It reduces cognitive load. On the other hand, comparisons and value statements are hard. There’s more cognitive load. So instead of Working software over comprehensive documentation, people say don’t bother to write documentation. And from comments often are used as a deodorant we get Code never lies, comments sometimes do and conventional wisdom saying comments are bad. It is known.

The thing is, Just like in Game Of Thrones, just because it is known, it isn’t necessarily true. Taking these value statements as binaries leads to the bigger problem of the things we know that just ain’t so.

Are comments a code smell? Yes. That just means they might be a bad thing. Are they a bad thing? They might be. It depends. Comments that just duplicate the code are a smell because the code and the comments don’t always change together. Which means you end up with stale comments. And those are a bad thing.

Comments that describe the code because the code is hard to understand are also a code smell. This is where the phrase comments are sweet smelling comes from. The comment is there because it smells good. It’s definitely a code smell. It makes the code easier to understand. It helps the next person who must look at the code.

But just like regular deodorant, it’s masking another smell. One that isn’t so good. The fact that the code in question can’t be easily understood is also a smell. It’s just been masked. So even though the comment is a good smell, it’s still an indication of a problem that ought to be addressed.

Other times, comments are there to explain why. They’re there to let future developers know some bit of arcane business knowledge or external factor that means the code needs to be this way. Or they might be there to explain what conditions might arise that would cause the code to be sub-optimal, or maybe even be invalidated. Or what needs to happen so that the code could be removed entirely. All of those are smells. Things that are right for now, but might need to be looked at again later.

So yes, comments are a code smell. It’s a sign that you need to look at the code and the comment and think about why they’re there and what should be done about it and when.

But, just like all the other code smells, it’s not an absolute. It doesn’t mean Thou shalt not write comments. It means that you need to think before you type.

Diffusion Of Responsibility

I’ve talked about ownership before. I’ve even mentioned diffusion of responsibility a couple of times. But I haven’t really talked about it. It’s time to talk about it.

I’m a grandfather now. Have been for almost a year. It’s amazing watching my granddaughter grow and learn. It’s also mazing watching my daughter learn about being a parent, just like I did many years ago. Unsurprisingly, we have many of the same opinions on how to raise a child, but it’s about 30 years later, so we’ve all learned a few things and advice has changed. If nothing else, car seats have changed a lot. The car seats we used back in the day weighed a lot less than the baby. The infant one my granddaughter used weighed more than she did.

Some things haven’t changed though. You need to watch toddlers all the time. They’re curious, and they’re fast. One thing my daughter is doing that is a really good idea is that when they’re out and about with she’s out and about with her daughter and friends, especially near water, the baby has a dedicated watcher. Everyone is thinking about her, and will act if they see a problem, but someone has the responsibility. Because diffusion of responsibility is a thing. If everyone is watching, then no one is watching.

Obviously, it’s not a binary thing. But if you know that everyone is watching, you allow yourself to get distracted. You don’t force yourself to pay attention when there’s something else pulling at you. Because you know others are watching, and they’ll notice if something happens and do something. Right? Probably, but not certainly. They might be thinking the same thing. Counting on someone else to be paying close attention.

The same thing applies to software design and development. Particularly to the non-functional requirements(NFRs). Everyone is responsible for them, in all parts of the software development lifecycle. When you’re developing a new feature or fixing a defect/bug/regression, you know that’s what you’re trying to do, so you focus on that. That does not mean, however, that you stop being responsible for the NFRs. You’re always responsible for them. And so is everyone else.

The thing is, software development is a socio-technical activity, and as we’ve seen, with us humans, the default is that if everyone is responsible, no one is. That means we need to be on guard for that bias, and work to ensure that it doesn’t let us get into bad habits.

There are some simple things you can do that can help. When you’re designing that new feature of fixing that defect, think about the future. Think about not just the specific problem, but the general class of problems. Can you make some potential issues impossible with your design? Like everything else, the answer to the question of “How much abstraction should I add here?” is “It Depends”. If you’ve been asked for a code review, either do it well and truly, or decline. You do your part by either doing the work or making it clear that someone else needs to do it. If you’re sending code out for review, review it yourself. Take the time to look at it with fresh eyes as someone else would look at it.

There are organization things you can do. Maintaining code quality is everyone’s responsibility, but you can have a team whose job is to make sure that people are doing that. That’s what my team at work does. We don’t add the quality. We don’t schedule the tasks to add quality. But we do look at things from a quality perspective. We let folks know where they’ve missed the mark, and we provide tools and training to help them do better.

Ownership and responsibility are important. Everyone can’t be responsible for everything, or nothing would get done. But that doesn’t mean that no one is responsible for some things. Some things are everyone’s responsibility.

It’s just that sometimes we need to be reminded that there are some things everyone is responsible for. And everyone needs to do. Those things are often called Norms. And that’s a discussion for another day.

Code Coverage Is Not Enough

Tales from the real word today. Code coverage is important. It can give you confidence. It can help you move faster. It can alert you to problems before they get exposed to your users. But it’s no guarantee of correctness.

Bad Tests

The first problem is that 100% code coverage does not in any way guarantee that the tests you’re using to measure your coverage are valid tests. Writing good unit tests is hard. A unit test that calls a function then asserts true and returns gives the same coverage as a well written unit test. There’s enough to say about the writing of good tests to fill up at least 3 more posts, so I’ll leave that for another day.

Transitive Coverage

Unit tests almost always have some transitive dependencies. Unless you’re careful, you’ll end up counting the coverage of those transitive dependencies as part of your code coverage. Because you aren’t trying to test those dependencies, only how you act with them, you get very little signal on the correctness of them. But since you’re just looking at “was this line executed?”, you get coverage. You don’t know if that coverage comes from direct testing, where the coverage means something, or from transitive coverage where it means nothing. That erodes a bit of confidence.

Branch Coverage

Another problem you’re going to have is complex branching statements and short circuiting logic. It’s part of writing good tests, but if you’re and’ing 4 booleans together there are 16 ways you need to test it. Add in some or’s and groupings, and it’s very easy to test both sides of a branch, but not really validate that you’ve covered all the possible cases. Another bit of confidence gone.

Integration Coverage and Emergent Behavior

But even if your unit tests are good, and you have 100% coverage of the source lines, you don’t necessarily have coverage of all the possible combinations of those things. As your system gets bigger, and the components interact in different ways, you get more and more emergent behavior. And your unit tests don’t validate ANY of those interactions. So you lose another bit of confidence,

But, But, But

If code coverage doesn’t guarantee correctness, why should you care? Why do ISO-26262 and MISRA talk about it? Why should you bother to measure or do anything about code coverage?

Because while code coverage doesn’t guarantee correctness, changes in code coverage very reliably tells you when something has changed. It helps guide you to where there is more work to be done. When you follow that guidance, you can improve your confidence in the correctness of your code. And that’s what makes code coverage valuable.

And one last thought on testing-based coverage. Don’t forget, testing can’t prove things are right. It just proves things aren’t wrong as your tests define wrong.