In looking at Table A.1, the ground rules for identifying the results of signed
integer comparisons become clear. Here’s a quick summary of the basic rules:
■■ Anytime ZF is set you know that the subtraction resulted in a zero,
which means that the operands are equal.
■■ When all three flags are zero, you know that the first operand is greater
than the second, because you have a positive result and no overflow.
■■ When there is a negative result and no overflow (SF=1 and OF=0), you
know that the second operand is larger than the first.
■■ When there is an overflow and a positive result, the second operand
must be larger than the first, because you essentially have a negative
result that is too small to be represented by the destination operand
(hence the overflow).
■■ When you have an overflow and a negative result, the first operand
must be larger than the second, because you essentially have a positive
result that is too large to be represented by the destination operand
(hence the overflow).While it is not generally necessary to memorizethe comparison outcome
tables (tables A.1 and A.2), it still makes sense to go over them and make sure
that you properly understand how each flag is used in the operand compari-
son process. This will be helpful in some cases while reversing when flags are
used in unconventional ways. Knowing how flags are set during comparison
and subtraction is very helpful for properly understanding logical sequences
and quickly deciphering their meaning.Unsigned Comparisons
Table A.2 demonstrates the behavior of the CMPinstruction when comparing
unsigned operands. Remember that just like table A.1, the following table also
applies to the SUBinstruction.Table A.2 Unsigned Subtraction Outcome Table for CMP and SUB Instructions (X repre-
sents the left operand, while Y represents the right operand)
RELATION
BETWEEN FLAGS
OPERANDS AFFECTED COMMENTS
X= Y CF = 0 ZF = 1 The two operands are equal, so the result is
zero.
X< Y CF = 1 ZF = 0 Yis larger than Xso the result is lower than
0, which generates an overflow (CF=1).
X > Y CF = 0 ZF = 0 Xis larger than Y, so the result is above zero,
and no overflow is generated (CF=0).482 Appendix A21_574817 appa.qxd 3/16/05 8:52 PM Page 482