Reversing : The Hacker's Guide to Reverse Engineering

After the 64-bit left-shifting function returns, you get into the following
code:

00402B1C ADD EAX,DWORD PTR SS:[EBP-190]

00402B22 MOV ECX,DWORD PTR SS:[EBP-18C]

00402B28 ADC ECX,EDX

00402B2A MOV DWORD PTR SS:[EBP-190],EAX

00402B30 MOV DWORD PTR SS:[EBP-18C],ECX

Figure 11.16 shows what this sequence does in mathematical notation.
Essentially, Defender is preparing a 64-bit integer that uniquely represents the
username string by taking each character and adding it at a unique bit position
in the 64-bit integer.
The function proceeds to perform a similar, but slightly less complicated
conversion on the serial number. Here, it just takes the 16 hexadecimal digits
and directly converts them into a 64-bit integer. Once it has that integer it calls
into 401EBC, pushing both 64-bit integers into the stack. At this point, you’re
hoping to find some kind of verification logic in 401EBCthat you can easily
understand. If so, you’ll have cracked Defender!

Validating User Information

Of course, 401EBCis also encrypted, but there’s something different about
this sequence. Instead of having a hard-coded decryption key for the XOR
operation or read it from a global variable, this function is calling into another
function (at 401D18) to obtain the key. Once 401D18returns, the function
stores its return value at [EBP-1C]where it is used during the decryption
process.

Figure 11.16 Equation used by Defender to convert username string to a 64-bit value.

Sum = (^) ΣCn × 2

Cn mod48

n = 0

len

Breaking Protections 407