Reverse Engineering for Beginners

CHAPTER 17. FLOATING-POINT UNIT CHAPTER 17. FLOATING-POINT UNIT

fld [ebp+a]

fld [ebp+b]

; current stack state: ST(0) - b; ST(1) - a

fxch st(1) ; this instruction swapping ST(1) and ST(0)

; current stack state: ST(0) - a; ST(1) - b

fucompp ; compare a and b and pop two values from stack, i.e., a and b

fnstsw ax ; store FPU status to AX

sahf ; load SF, ZF, AF, PF, and CF flags state from AH

setnbe al ; store 1 to AL, if CF=0 and ZF=0

test al, al ; AL==0?

jz short loc_8048453 ; yes

fld [ebp+a]

jmp short locret_8048456

loc_8048453:
fld [ebp+b]

locret_8048456:
leave
retn
d_max endp

FUCOMPPis almost likeFCOM, but pops both values from the stack and handles “not-a-numbers” differently.

A bit aboutnot-a-numbers.

The FPU is able to deal with special values which arenot-a-numbersorNaNs^17. These are infinity, result of division by 0, etc.
Not-a-numbers can be “quiet” and “signaling”. It is possible to continue to work with “quiet” NaNs, but if one tries to do any
operation with “signaling” NaNs, an exception is to be raised.

FCOMraising an exception if any operand isNaN.FUCOMraising an exception only if any operand is a signalingNaN(SNaN).

The next instruction isSAHF(Store AH into Flags) —this is a rare instruction in code not related to the FPU. 8 bits from AH
are moved into the lower 8 bits of the CPU flags in the following order:
7 6 4 2 0

SFZF AF PF CF

Let’s recall thatFNSTSWmoves the bits that interest us (C3/C2/C0) intoAHand they are in positions 6, 2, 0 of theAHregister:
6 2 1 0

C3 C2C1C0

In other words, thefnstsw ax / sahfinstruction pair movesC3/C2/C0intoZF,PFandCF.

Now let’s also recall the values ofC3/C2/C0in different conditions:

• Ifais greater thanbin our example, thenC3/C2/C0are to be set to: 0, 0, 0.


• ifais less thanb, then the bits are to be set to: 0, 0, 1.


• Ifa=b, then: 1, 0, 0.

In other words, these states of the CPU flags are possible after threeFUCOMPP/FNSTSW/SAHFinstructions:

• Ifa>b, the CPU flags are to be set as:ZF=0, PF=0, CF=0.


• Ifa<b, then the flags are to be set as:ZF=0, PF=0, CF=1.


• And ifa=b, then:ZF=1, PF=0, CF=0.

Depending on the CPU flags and conditions,SETNBEstores 1 or 0 to AL. It is almost the counterpart ofJNBE, with the
exception thatSETcc^18 stores 1 or 0 inAL, butJccdoes actually jump or not.SETNBEstores 1 only ifCF=0andZF=0. If
it is not true, 0 is to be stored intoAL.

Only in one case bothCFandZFare 0: ifa>b.

Then 1 is to be stored toAL, the subsequentJZis not to be triggered and the function will return _a. In all other cases, _b
is to be returned.

(^17) wikipedia.org/wiki/NaN
(^18) cciscondition code