Active Antidebugger Techniques
Because a large part of the reversing process often takes place inside a debug-
ger, it is sometimes beneficial to incorporate special code in the program that
prevents or complicates the process of stepping through the program and
placing breakpoints in it. Antidebugger techniques are particularly effective
when combined with code encryption because encrypting the program forces
reversers to run it inside a debugger in order to allow the program to decrypt
itself. As discussed earlier, it is sometimes possible to unpack programs auto-
matically using unpackers without running them, but it is possible to create a
code encryption scheme that make it impossible to automatically unpack the
encrypted executable.
Throughout the years there have been dozens of antidebugger tricks, but it’s
important to realize that they are almost always platform-specific and depend
heavily on the specific operating system on which the software is running.
Antidebugger tricks are also risky, because they can sometimes generate false
positives and cause the program to malfunction even though no debugger is
present. The same is not true for code obfuscation, in which the program typi-
cally grows in footprint or decreases in runtime performance, but the costs can
be calculated in advance, and there are no unpredictable side effects.
The rest of this section explains some debugger basics which are necessary
for understanding these antidebugger tricks, and proceeds to discuss specific
antidebugging tricks that are reasonably effective and are compatible with NT-
based operating systems.
Debugger Basics
To understand some of the antidebugger tricks that follow a basic understand-
ing of how debuggers work is required. Without going into the details of how
user-mode and kernel-mode debuggers attach into their targets, let’s discuss
how debuggers pause and control their debugees. When a user sets a break-
point on an instruction, the debugger usually replaces that instruction with an
int 3instruction. The int 3instruction is a special breakpoint interrupt that
notifies the debugger that a breakpoint has been reached. Once the debugger
is notified that the int 3has been reached, it replaces the int 3with the
original instruction from the program and freezes the program so that the
operator (typically a software developer) can inspect its state.
An alternative method of placing breakpoints in the program is to use hard-
ware breakpoints. A hardware breakpoint is a breakpoint that the processor itself
manages. Hardware breakpoints don’t modify anything in the target pro-
gram—the processor simply knows to break when a specific memory address
is accessed. Such a memory address could either be a data address accessed by
Antireversing Techniques 331