(^160) 97 Things Every Programmer Should Know
Test for Required
Behavior, Not
Incidental Behavior
Kevlin Henney
A OMMONC PiTFALL iN TESTiNG is to assume that exactly what an imple-
mentation does is precisely what you want to test for. At first glance, this
sounds more like a virtue than a pitfall. Phrased another way, however, the
issue becomes more obvious: a common pitfall in testing is to hardwire tests
to the specifics of an implementation, where those specifics are incidental and
have no bearing on the desired functionality.
When tests are hardwired to implementation incidentals, changes to the imple-
mentation that are actually compatible with the required behavior may cause
tests to fail, leading to false positives. Programmers typically respond either
by rewriting the test or by rewriting the code. Assuming that a false positive is
actually a true positive is often a consequence of fear, uncertainty, or doubt. It
has the effect of raising the status of incidental behavior to required behavior.
In rewriting a test, programmers either refocus the test on the required behav-
ior (good) or simply hardwire it to the new implementation (not good). Tests
need to be sufficiently precise, but they also need to be accurate.
For example, in a three-way comparison, such as Java’s String.compareTo or C’s
strcmp, the requirements on the result are that it is negative if the lefthand side
is less than the right, positive if the lefthand side is greater than the right, and
zero if they are considered equal. This style of comparison is used in many
APIs, including the comparator for C’s qsort function and compareTo in Java’s
Comparable interface. Although the specific values –1 and +1 are commonly used