Reversing : The Hacker's Guide to Reverse Engineering

Pure Arithmetic Implementations

Certain logical statements can be converted directly into a series of arithmetic

operations, involving no conditional execution whatsoever. These are elegant

mathematical tricks that allow compilers to translate branched logic in the

source code into a simple sequence of arithmetic operations. Consider the fol-

lowing code:

mov eax, [ebp - 10]

and eax, 0x00001000

neg eax

sbb eax, eax

neg eax

ret

The preceding compiler-generated code snippet is quite common in IA-32

programs, and many reversers have a hard time deciphering its meaning. Con-

sidering the popularity of these sequences, you should go over this sample

and make sure you understand how it works.

The code starts out with a simple logical ANDof a local variable with

0x00001000, storing the result into EAX(the ANDinstruction always sends

the result to the first, left-hand operand). You then proceed to a NEGinstruc-

tion, which is slightly less common. NEGis a simple negation instruction,

which reverses the sign of the operand—this is sometimes called two’s com-

plement. Mathematically, NEGperforms a simple

Result = -(Operand);

operation. The interesting part of this sequence is the SBBinstruction. SBBis a

subtraction with borrow instruction. This means that SBBtakes the second

(right-hand) operand and adds the value of CF to it and then subtracts the

result from the first operand. Here’s a pseudocode for SBB:

Operand1 = Operand1 – (Operand2 + CF);

Notice that in the preceding sample SBBwas used on a single operand. This

means that SBBwill essentially subtract EAXfrom itself, which of course is a

mathematically meaningless operation if you disregard CF. Because CF is

added to the second operand, the result will depend solely on the value of CF.

If CF == 1, EAXwill become –1. If CF == 0, EAXwill become zero. It should

be obvious that the value of EAXafter the first NEGwas irrelevant. It is immedi-

ately lost in the following SBBbecause it subtracts EAXfrom itself. This raises

the question of whydid the compiler even bother with the NEGinstruction?

The Intel documentation states that beyond reversing the operand’s sign,

NEGwill also set the value of CF based on the value of the operand. If the

operand is zero when NEGis executed, CF will be set to zero. If the operand is

510 Appendix A

21_574817 appa.qxd 3/16/05 8:54 PM Page 510