Reverse Engineering for Beginners

CHAPTER 17. FLOATING-POINT UNIT CHAPTER 17. FLOATING-POINT UNIT

17.7 Comparison example

Let’s try this:

#include <stdio.h>

double d_max (double a, double b)
{
if (a>b)
return a;

return b;
};

int main()
{
printf ("%f\n", d_max (1.2, 3.4));
printf ("%f\n", d_max (5.6, -4));
};

Despite the simplicity of the function, it will be harder to understand how it works.

17.7.1 x86

Non-optimizing MSVC

MSVC 2010 generates the following:

Listing 17.10: Non-optimizing MSVC 2010

PUBLIC _d_max
_TEXT SEGMENT
_a$ = 8 ; size = 8
_b$ = 16 ; size = 8
_d_max PROC
push ebp
mov ebp, esp
fld QWORD PTR _b$[ebp]

; current stack state: ST(0) = _b
; compare _b (ST(0)) and _a, and pop register

fcomp QWORD PTR _a$[ebp]

; stack is empty here

fnstsw ax

test ah, 5

jp SHORT $LN1@d_max

; we are here only if a>b

fld QWORD PTR _a$[ebp]
jmp SHORT $LN2@d_max
$LN1@d_max:
fld QWORD PTR _b$[ebp]
$LN2@d_max:
pop ebp
ret 0
_d_max ENDP

So,FLDloads_bintoST(0).

FCOMPcompares the value inST(0)with what is in_aand setsC3/C2/C0bits in FPU status word register, accordingly.
This is a 16-bit register that reflects the current state of the FPU.

After the bits are set, theFCOMPinstruction also pops one variable from the stack. This is what distinguishes it fromFCOM,
which is just compares values, leaving the stack in the same state.