MD5 (MD stands for message-digest) is a highly popular cryptographic hash-
ing algorithm that produces a long (128-bit) hash or checksum from a variable-
length message. This hash can later be used to uniquely identify the specific
message. Two basic properties of MD5 and other cryptographic hashes are that
it is extremely unlikely that there would ever be two different messages that pro-
duce the same hash and that it is virtually impossible to create a message that
will generate a predetermined hash value.
With this information, let’s proceed to determine the nature of the data that
Cryptex is hashing. This can be easily gathered by inspecting the call to
CryptHashData. According to the MSDN, the second parameter passed to
CryptHashDatais the data being hashed. In Listing 6.4, Cryptex is passing
EDX, which was earlier loaded from [ESP+C]. The third parameter is the
buffer length, which is set to 0x14(20 bytes). A quick look at the buffer pointer
to by [ESP+C]shows the following.
0012F5E8 77 03 BE 9F EC CA 20 05 D0 D6 DF FB A2 CF 55 4B
0012F5F8 81 41 C0 FENothing obvious here—this isn’t text or anything, just more unrecognized
data. The next thing Cryptex does is call CryptGetHashParamon the hash
object, with the value 2 in the second parameter. A quick search through
WinCrypt.Hshows that the value 2 stands for HP_HASHVAL. This means that
Cryptex is asking for the actual hash value (that’s the MD5 result for those 20
bytes from 0012F5E8). The third parameter passed to CryptGetHashParam
tells the function where to write the hash value. Guess what? It’s being written
into 00405038 , the global variable that was used earlier for checking whether
the password matches.
To summarize, Cryptex is apparently hashing unknown, nontextual data
using the MD5 hashing algorithm, and is writing the result into a global vari-
able. The contents of this global variable are later compared against a value
stored in the Cryptex archive file. If it isn’t identical, Cryptex reports an incor-
rect password. It is obvious that the data that is being hashed in the function
from Listing 6.4 is clearly somehow related to the password that was typed. We
just don’t understand the connection. The unknown data that was hashed in
this function was passed as a parameter from the calling function.
Hashing the Password
At this point you’re probably a bit at a loss regarding the origin of the buffer,
you just hashed in Listing 6.4. In such cases, it is usually best to simply trace
back in the program until you find the origin of that buffer. In this case, the
hashed buffer came from the calling function, at 00402300. This function is
shown in Listing 6.5.
Deciphering File Formats 213