Reversing : The Hacker's Guide to Reverse Engineering

that stores the currently typed password. This is the variable at 00405038
against which the header data was compared in Listing 6.3. In OllyDbg, a mem-
ory breakpoint can be set by opening the address ( 00405038 ) in the Dump win-
dow, right-clicking the address, and selecting Breakpoint➪Hardware, On
write➪Dword. Keep in mind that you must restart the program before you do
this because at the point where the bad password message is being printed this
variable has already been initialized.
Restart the program, place a hardware breakpoint on 00405038 , and let the
program run (with the same set of command-line parameters). The debugger
breaks somewhere inside RSAENH.DLL, the Microsoft Enhanced Cryptographic
Provider. Why is the Microsoft Enhanced Cryptographic Provider writing into a
global variable from Cryptex.exe? Probably because Cryptex.EXEhad sup-
plied the address of that global variable. Let’s look at the stack and try to trace
back and find the call made from Cryptex to the encryption engine. In tracing
back through the stack in the Stack Window, you can see that we are currently
running inside the CryptGetHashParamAPI, which was called from a func-
tion inside Cryptex. Listing 6.4 shows the code for this function.

00402280 MOV ECX,DS:[405048]

00402286 SUB ESP,8

00402289 LEA EAX,SS:[ESP]

0040228C PUSH EAX

0040228D PUSH 0

0040228F PUSH 0

00402291 PUSH 8003

00402296 PUSH ECX

00402297 CALL DS:[<&ADVAPI32.CryptCreateHash>]

0040229D TEST EAX,EAX

0040229F JE SHORT cryptex.004022C2

004022A1 MOV EDX,SS:[ESP+C]

004022A5 MOV EAX,SS:[ESP]

004022A8 PUSH 0

004022AA PUSH 14

004022AC PUSH EDX

004022AD PUSH EAX

004022AE CALL DS:[<&ADVAPI32.CryptHashData>]

004022B4 TEST EAX,EAX

004022B6 MOV ECX,SS:[ESP]

004022B9 JNZ SHORT cryptex.004022C8

004022BB PUSH ECX

004022BC CALL DS:[<&ADVAPI32.CryptDestroyHash>]

004022C2 XOR EAX,EAX

004022C4 ADD ESP,8

004022C7 RETN

Listing 6.4 Function in Cryptex that calls into the cryptographic service provider—the 16-
byte password-identifier value is written from within this function. (continued)

Deciphering File Formats 211