Time Passes
Hot Take. Unit tests should pass, regardless of what day you run them. Time is hard. It has a way of passing when you’re not even thinking about it. When you’re writing simulations (or unit tests, which can be thought of as simulations of some small aspect of your code) one of the most important things to do is control time. As a general rule, unless you’re measuring performance or displaying/logging the current time, you probably shouldn’t be using your language’s equivalent of Time.Now(). In fact, even in those cases, I’ll assert that you shouldn’t be calling it directly. You should at least be using some kind of dependency injection, if not a whole façade1.
The other day I was dealing with an error in unit tests. A test on a function that hadn’t been changed in a while started to fail. I was able to reproduce the error locally, and wanted to find out which change caused it. I tried to track it down, using git bisect to help me do it, but I wasn’t able to. Between the time I got the error to happen locally and when I got back to dealing with it, the error magically went away.
Heisenbugs are a terrible thing. The tests pass, but the bug is just sitting there waiting to bite you. It’s never a fun time when you have to find and fix one. One nice thing about them, to use the term nice loosely, is that they’re usually related to one of a few things. The environment (things like ENV variables, files on disk, current working directory), load on the machine (disk space, CPU or network load), multi-threading, or time.
In this case it was time. But not in the code. The code was just comparing the difference in working days between two dates. It was, in fact, correct. It used a calendar and the two dates and gave the right answer.
Instead, this was in fact a kind of Schrödinger’s test. Depending on when the test was run, sometimes it passed, and sometimes it failed. The test was checking that the number of working days between now and three days ago was always at least one.
That seems reasonable. Or at least on the surface, that seems reasonable. Since working days are Monday to Friday, with Saturday and Sunday being weekends, there are never more than two non-working days in any three day period, so there’s always at least one working day.
And that’s how the test worked. It looked something like
today = time.Now()
three_days_ago = today.add(‘day’, -3)
result = working_days_between(today, three_days_ago)
assert result >= 1
The problem was, the test forgot about the fact that it’s not always true, Like on a three or four day weekend. Like Memorial Day in the US. Run the test on most Tuesdays and it passes. The number of working days between Tuesday and the preceding Saturday is one (the Monday between them). But run it on the Tuesday right after Memorial day and the number of working days between them is zero. That Monday is not a working day. The function did the right thing. It normally returned 1, but that day it returned zero. And the test failed2.
This is actually a hard function to test correctly. Any day can be a holiday. It’s a little better defined for official holidays but add in company holidays, religious holidays, and personal holidays, and it’s un-knowable at test time. There are just too many variables. If you don’t tell the function when holidays are you either have to know when you’re writing the test or find them out at test time.
The most robust way to test this is to change the function to take a calendar, then in the test pass in not just the two days, but the calendar that should be used. And then calculate how many working days there are between the two dates in the test. Then, assert that the return value is exactly the same. Then figure out the edge cases and use boundary value analysis to make sure you test all of them.
And by the way, don’t forget that your calendar will change over time, so when you ask the question, how many working days are there between two dates, you need to think about when the question is being asked and know the calendar that was being used at that time. Just in case you didn’t think this was complicated enough
