Reverse Engineering for Beginners

(avery) #1

CHAPTER 79. “QR9”: RUBIK’S CUBE INSPIRED AMATEUR CRYPTO-ALGORITHM CHAPTER 79. “QR9”: RUBIK’S CUBE INSPIRED AMATEUR CRYPTO-ALGORITHM


Open the file and report if an error occurs:


.text:00541320 mov eax, [esp+Str]
.text:00541324 push ebp
.text:00541325 push offset Mode ; "rb"
.text:0054132A push eax ; Filename
.text:0054132B call _fopen ; open file
.text:00541330 mov ebp, eax
.text:00541332 add esp, 8
.text:00541335 test ebp, ebp
.text:00541337 jnz short loc_541348
.text:00541339 push offset Format ; "Cannot open input file!\n"
.text:0054133E call _printf
.text:00541343 add esp, 4
.text:00541346 pop ebp
.text:00541347 retn
.text:00541348
.text:00541348 loc_541348:


Get the file size viafseek()/ftell():


.text:00541348 push ebx
.text:00541349 push esi
.text:0054134A push edi
.text:0054134B push 2 ; Origin
.text:0054134D push 0 ; Offset
.text:0054134F push ebp ; File


; move current file position to the end
.text:00541350 call _fseek
.text:00541355 push ebp ; File
.text:00541356 call _ftell ; get current file position
.text:0054135B push 0 ; Origin
.text:0054135D push 0 ; Offset
.text:0054135F push ebp ; File
.text:00541360 mov [esp+2Ch+Str], eax


; move current file position to the start
.text:00541364 call _fseek


This fragment of code calculates the file size aligned on a 64-byte boundary. This is because this cryptographic algorithm
works with only 64-byte blocks. The operation is pretty straightforward: divide the file size by 64, forget about the remainder
and add 1, then multiply by 64. The following code removes the remainder as if the value was already divided by 64 and
adds 64. It is almost the same.


.text:00541369 mov esi, [esp+2Ch+Str]
; reset all lowest 6 bits
.text:0054136D and esi, 0FFFFFFC0h
; align size to 64-byte border
.text:00541370 add esi, 40h


Allocate buffer with aligned size:


.text:00541373 push esi ; Size
.text:00541374 call _malloc


Call memset(), e.g., clear the allocated buffer^2.


.text:00541379 mov ecx, esi
.text:0054137B mov ebx, eax ; allocated buffer pointer -> to EBX
.text:0054137D mov edx, ecx
.text:0054137F xor eax, eax
.text:00541381 mov edi, ebx
.text:00541383 push ebp ; File
.text:00541384 shr ecx, 2


(^2) malloc() + memset() could be replaced by calloc()

Free download pdf