check that the function has not been modified. This method is not only effec-
tive against debuggers, but also against code patching (see Chapter 11), but
has the downside that constantly recalculating checksums is a relatively
expensive operation.
There are several workarounds for this problem; it all boils down to employ-
ing a clever design. Consider, for example, a program that has 10 highly sensi-
tive functions that are called while the program is loading (this is a common
case with protected applications). In such a case, it might make sense to have
each function verify its own checksum prior to returning to the caller. If the
checksum doesn’t match, the function could take an inconspicuous (so that
reversers don’t easily spot it) detour that would eventually lead to the termi-
nation of the program or to some kind of unusual program behavior that
would be very difficult for the attacker to diagnose. The benefit of this
approach is that it doesn’t add much execution time to the program because
only the specific functions that are considered to be sensitive are affected.
Note that this technique doesn’t detect or prevent hardware breakpoints,
because such breakpoints don’t modify the program code in any way.Confusing Disassemblers
Fooling disassemblers as a means of preventing or inhibiting reversers is not a
particularly robust approach to antireversing, but it is popular none the less.
The strategy is quite simple. In processor architectures that use variable-length
instructions, such as IA-32 processors, it is possible to trick disassemblers into
incorrectly treating invalid data as the beginning of an instruction. This causes
the disassembler to lose synchronization and disassemble the rest of the code
incorrectly until it resynchronizes.
Before discussing specific techniques, I would like to briefly remind you of
the two common approaches to disassembly (discussed in Chapter 4). A linear
sweep is the trivial approach that simply disassembles instruction sequentially
in the entire module. Recursive traversal is the more intelligent approach
whereby instructions are analyzed by traversing instructions while following
the control flow instructions in the program, so that when the program
branches to a certain address, disassembly also proceeds at that address.
Recursive traversal disassemblers are more reliable and are far more tolerant
of various antidisassembly tricks.
Let’s take a quick look at the reversing tools discussed in this book and see
which ones actually use recursive traversal disassemblers. This will help you
predict the effect each technique is going to have on the most common tools.
Table 10.1 describes the disassembly technique employed in the most common
reversing tools.336 Chapter 10