Assembly Language for Beginners

1 Code Patterns

2 Important fundamentals

3 Slightly more advanced examples

4 Java

5 Finding important/interesting stuff in the code

6 OS-specific

7 Tools

8 Case studies

9 Examples of reversing proprietary file formats

10 Dynamic binary instrumentation

11 Other things

12 Books/blogs worth reading

13 Communities

Afterword

Appendix

Acronyms used

Glossary

Index

1 Code Patterns Contents

1.1 The method

1.2 Some basics.

1.2.1 A short introduction to the CPU

1.2.2 Numeral Systems

1.2.3 Converting From One Radix To Another

1.3 An Empty Function.

1.3.1 x86

1.3.2 ARM.

1.3.3 MIPS.

1.3.4 Empty Functions in Practice.

1.4 Returning Values

1.4.1 x86

1.4.2 ARM.

1.4.3 MIPS.

1.5 Hello, world!.

1.5.1 x86

1.5.2 x86-64

1.5.3 GCC—one more thing.

1.5.4 ARM.

1.5.5 MIPS.

1.5.6 Conclusion.

1.5.7 Exercises.

1.6 Function prologue and epilogue

1.6.1 Recursion

1.7 Stack

1.7.1 Why does the stack grow backwards?

1.7.2 What is the stack used for?.

1.7.3 A typical stack layout.

1.7.4 Noise in stack.

1.7.5 Exercises.

1.8 printf() with several arguments.

1.8.1 x86

1.8.2 ARM.

1.8.3 MIPS.

1.8.4 Conclusion.

1.8.5 By the way

1.9 scanf().

1.9.1 Simple example

1.9.2 Popular mistake

1.9.3 Global variables

1.9.4 scanf()

1.9.5 Exercise

1.10 Accessing passed arguments

1.10.1 x86

1.10.2 x64

1.10.3 ARM

1.10.4 MIPS.

1.11 More about results returning

1.11.1 Attempt to use the result of a function returningvoid

1.11.2 What if we do not use the function result?

1.11.3 Returning a structure.

1.12 Pointers.

1.12.1 Swap input values. CONTENTS

1.12.2 Returning values.

1.13 GOTO operator

1.13.1 Dead code.

1.13.2 Exercise

1.14 Conditional jumps

1.14.2 Calculating absolute value.

1.14.3 Ternary conditional operator

1.14.4 Getting minimal and maximal values.

1.14.5 Conclusion.

1.14.6 Exercise

1.15 switch()/case/default

1.15.1 Small number of cases.

1.15.2 A lot of cases

1.15.3 When there are severalcasestatements in one block.

1.15.4 Fall-through

1.15.5 Exercises.

1.16 Loops

1.16.2 Memory blocks copying routine

1.16.3 Condition check

1.16.4 Conclusion.

1.16.5 Exercises.

1.17 More about strings.

1.17.1 strlen()

1.17.2 Boundaries of strings.

1.18 Replacing arithmetic instructions to other ones

1.18.1 Multiplication

1.18.2 Division.

1.18.3 Exercise

1.19 Floating-point unit

1.19.1 IEEE

1.19.2 x86

1.19.3 ARM, MIPS, x86/x64 SIMD

1.19.4 C/C++

1.19.6 Passing floating point numbers via arguments.

1.19.7 Comparison example.

1.19.8 Some constants

1.19.9 Copying.

1.19.10 Stack, calculators and reverse Polish notation

1.19.11 80 bits?.

1.19.12 x64.

1.19.13 Exercises.

1.20 Arrays

1.20.2 Buffer overflow.

1.20.3 Buffer overflow protection methods.

1.20.4 One more word about arrays

1.20.5 Array of pointers to strings

1.20.6 Multidimensional arrays

1.20.7 Pack of strings as a two-dimensional array

1.20.8 Conclusion.

1.21 By the way.

1.21.1 Exercises.

1.22 Manipulating specific bit(s)

1.22.1 Specific bit checking

1.22.2 Setting and clearing specific bits

1.22.3 Shifts

1.22.4 Setting and clearing specific bits: FPU^1 example

1.22.5 Counting bits set to 1.

1.22.6 Conclusion.

1.22.7 Exercises. CONTENTS

1.23 Linear congruential generator.

1.23.1 x86

1.23.2 x64

1.23.3 32-bit ARM.

1.23.4 MIPS.

1.23.5 Thread-safe version of the example.

1.24 Structures

1.24.1 MSVC: SYSTEMTIME example.

1.24.2 Let’s allocate space for a structure using malloc()

1.24.3 UNIX: struct tm.

1.24.4 Fields packing in structure.

1.24.5 Nested structures

1.24.6 Bit fields in a structure

1.24.7 Exercises.

1.25 Unions.

1.25.1 Pseudo-random number generator example

1.25.2 Calculating machine epsilon

1.26 FSCALE replacement.

1.26.1 Fast square root calculation.

1.27 Pointers to functions.

1.27.1 MSVC

1.27.2 GCC

1.27.3 Danger of pointers to functions

1.28 64-bit values in 32-bit environment.

1.28.1 Returning of 64-bit value.

1.28.2 Arguments passing, addition, subtraction.

1.28.3 Multiplication, division

1.28.4 Shifting right

1.28.5 Converting 32-bit value into 64-bit one

1.29 SIMD.

1.29.1 Vectorization

1.29.2 SIMDstrlen()implementation

1.30 64 bits.

1.30.1 x86-64

1.30.2 ARM

1.30.3 Float point numbers.

1.30.4 64-bit architecture criticism.

1.31 Working with floating point numbers using SIMD.

1.31.2 Passing floating point number via arguments

1.31.3 Comparison example.

1.31.4 Calculating machine epsilon: x64 and SIMD

1.31.5 Pseudo-random number generator example revisited

1.31.6 Summary.

1.32 ARM-specific details

1.32.1 Number sign (#) before number.

1.32.2 Addressing modes.

1.32.3 Loading a constant into a register.

1.32.4 Relocs in ARM64

1.33 MIPS-specific details.

1.33.1 Loading a 32-bit constant into register.

1.33.2 Further reading about MIPS

2 Important fundamentals

2.1 Integral datatypes.

2.1.1 Bit

2.1.2 Nibble AKA nybble.

2.1.3 Byte.

2.1.4 Wide char

2.1.5 Signed integer vs unsigned.

2.1.6 Word

2.1.7 Address register

2.1.8 Numbers.

2.2 Signed number representations CONTENTS

2.2.1 Using IMUL over MUL.

2.2.2 Couple of additions about two’s complement form.

2.3 Integer overflow

2.4 AND

2.4.1 Checking if a value is on 2 nboundary

2.4.2 KOI-8R Cyrillic encoding

2.5 AND and OR as subtraction and addition

2.5.1 ZX Spectrum ROM text strings.

2.6 XOR (exclusive OR)

2.6.1 Everyday speech.

2.6.2 Encryption.

2.6.3 RAID

2.6.4 XOR swap algorithm

2.6.5 XOR linked list

2.6.6 Zobrist hashing / tabulation hashing

2.6.7 By the way

2.6.8 AND/OR/XOR as MOV.

2.7 Population count

2.8 Endianness.

2.8.1 Big-endian.

2.8.2 Little-endian.

2.8.3 Example

2.8.4 Bi-endian.

2.8.5 Converting data

2.9 Memory.

2.10 CPU.

2.10.1 Branch predictors

2.10.2 Data dependencies

2.11 Hash functions

2.11.1 How do one-way functions work?

3 Slightly more advanced examples

3.1 Double negation

3.2 strstr() example

3.3 Temperature converting

3.3.1 Integer values

3.3.2 Floating-point values

3.4 Fibonacci numbers.

3.4.1 Example #1.

3.4.2 Example #2.

3.4.3 Summary

3.5 CRC32 calculation example

3.6 Network address calculation example

3.6.1 calc_network_address()

3.6.2 form_IP().

3.6.3 print_as_IP().

3.6.4 form_netmask() and set_bit().

3.6.5 Summary

3.7 Loops: several iterators

3.7.1 Three iterators

3.7.2 Two iterators

3.7.3 Intel C++ 2011 case

3.8 Duff’s device

3.8.1 Should one use unrolled loops?

3.9 Division using multiplication

3.9.1 x86

3.9.2 How it works

3.9.3 ARM.

3.9.4 MIPS.

3.9.5 Exercise

3.10 String to number conversion (atoi())

3.10.2 A slightly advanced example. CONTENTS

3.10.3 Exercise

3.11 Inline functions

3.11.1 Strings and memory functions

3.12 C99 restrict.

3.13 Branchlessabs()function

3.13.1 Optimizing GCC 4.9.1 x64.

3.13.2 Optimizing GCC 4.9 ARM64

3.14 Variadic functions.

3.14.1 Computing arithmetic mean

3.14.2vprintf()function case

3.14.3 Pin case.

3.14.4 Format string exploit

3.15 Strings trimming

3.15.1 x64: Optimizing MSVC

3.15.2 x64: Non-optimizing GCC 4.9.1

3.15.3 x64: Optimizing GCC 4.9.1

3.15.4 ARM64: Non-optimizing GCC (Linaro) 4.9

3.15.5 ARM64: Optimizing GCC (Linaro) 4.9.

3.15.6 ARM: Optimizing Keil 6/2013 (ARM mode).

3.15.7 ARM: Optimizing Keil 6/2013 (Thumb mode)

3.15.8 MIPS.

3.16 toupper() function

3.16.1 x64

3.16.2 ARM

3.16.3 Using bit operations.

3.16.4 Summary.

3.17 Obfuscation

3.17.1 Text strings

3.17.2 Executable code

3.17.3 Virtual machine / pseudo-code.

3.17.4 Other things to mention

3.17.5 Exercise

3.18 C++

3.18.1 Classes

3.18.2 ostream.

3.18.3 References.

3.18.4 STL.

3.18.5 Memory.

3.19 Negative array indices

3.19.1 Addressing string from the end.

3.19.2 Addressing some kind of block from the end.

3.19.3 Arrays started at 1.

3.20 Packing 12-bit values into array

3.20.1 Introduction

3.20.2 Data structure

3.20.3 The algorithm.

3.20.4 The C/C++ code

3.20.5 How it works

3.20.6 Optimizing GCC 4.8.2 for x86-64.

3.20.7 Optimizing Keil 5.05 (Thumb mode).

3.20.8 Optimizing Keil 5.05 (ARM mode)

3.20.9 (32-bit ARM) Comparison of code density in Thumb and ARM modes

3.20.10 Optimizing GCC 4.9.3 for ARM64.

3.20.11 Optimizing GCC 4.4.5 for MIPS

3.20.12 Difference from the real FAT12

3.20.13 Exercise.

3.20.14 Summary.

3.20.15 Conclusion

3.21 More about pointers.

3.21.1 Working with addresses instead of pointers.

3.21.2 Passing values as pointers; tagged unions

3.21.3 Pointers abuse in Windows kernel.

3.21.4 Null pointers.

3.21.5 Array as function argument. CONTENTS

3.21.6 Pointer to function.

3.21.7 Pointer as object identificator.

3.22 Loop optimizations.

3.22.1 Weird loop optimization

3.22.2 Another loop optimization.

3.23 More about structures.

3.23.1 Sometimes a C structure can be used instead of array

3.23.2 Unsized array in C structure.

3.23.3 Version of C structure.

3.23.4 High-score file in “Block out” game and primitive serialization

3.24 memmove() and memcpy()

3.24.1 Anti-debugging trick.

3.25 setjmp/longjmp

3.26 Other weird stack hacks

3.26.1 Accessing arguments/local variables of caller

3.26.2 Returning string

3.27 OpenMP.

3.27.1 MSVC

3.27.2 GCC

3.28 Another heisenbug.

3.29 Windows 16-bit

3.29.1 Example#1

3.29.2 Example #2.

3.29.3 Example #3.

3.29.4 Example #4.

3.29.5 Example #5.

3.29.6 Example #6.

4 Java

4.1 Java

4.1.1 Introduction.

4.1.2 Returning a value

4.1.3 Simple calculating functions

4.1.4 JVM^3 memory model

4.1.5 Simple function calling.

4.1.6 Calling beep().

4.1.7 Linear congruential PRNG

4.1.8 Conditional jumps

4.1.9 Passing arguments

4.1.10 Bitfields.

4.1.11 Loops

4.1.12 switch().

4.1.13 Arrays.

4.1.14 Strings

4.1.15 Exceptions.

4.1.16 Classes

4.1.17 Simple patching

4.1.18 Summary.

5 Finding important/interesting stuff in the code

5.1 Identification of executable files

5.1.1 Microsoft Visual C++.

5.1.2 GCC.

5.1.3 Intel Fortran.

5.1.4 Watcom, OpenWatcom.

5.1.5 Borland.

5.1.6 Other known DLLs.

5.2 Communication with outer world (function level)

5.3 Communication with the outer world (win32).

5.3.1 Often used functions in the Windows API

5.3.2 Extending trial period.

5.3.3 Removing nag dialog box CONTENTS

5.3.4 tracer: Intercepting all functions in specific module

5.4 Strings

5.4.1 Text strings

5.4.2 Finding strings in binary

5.4.3 Error/debug messages.

5.4.4 Suspicious magic strings.

5.5 Calls to assert().

5.6 Constants

5.6.1 Magic numbers.

5.6.2 Specific constants.

5.6.3 Searching for constants

5.7 Finding the right instructions

5.8 Suspicious code patterns.

5.8.1 XOR instructions.

5.8.2 Hand-written assembly code

5.9 Using magic numbers while tracing.

5.10 Loops

5.10.1 Some binary file patterns

5.10.2 Memory “snapshots” comparing.

5.11 ISA^5 detection.

5.11.1 Incorrectly disassembled code

5.11.2 Correctly disassembled code

5.12 Text strings right in the middle of compressed data.

5.13 Other things

5.13.1 General idea.

5.13.2 Order of functions in binary code

5.13.3 Tiny functions.

5.13.4 C++.

6 OS-specific

6.1 Arguments passing methods (calling conventions)

6.1.1 cdecl

6.1.2 stdcall

6.1.3 fastcall

6.1.4 thiscall

6.1.5 x86-64

6.1.6 Return values offloatanddoubletype

6.1.7 Modifying arguments.

6.1.8 Taking a pointer to function argument.

6.2 Thread Local Storage

6.2.1 Linear congruential generator revisited

6.3 System calls (syscall-s).

6.3.1 Linux

6.3.2 Windows.

6.4 Linux

6.4.1 Position-independent code

6.4.2LD_PRELOADhack in Linux

6.5 Windows NT

6.5.1 CRT (win32).

6.5.2 Win32 PE.

6.5.3 Windows SEH.

6.5.4 Windows NT: Critical section

7 Tools

7.1 Binary analysis

7.1.1 Disassemblers

7.1.2 Decompilers.

7.1.3 Patch comparison/diffing.

7.2 Live analysis.

7.2.1 Debuggers.

7.2.2 Library calls tracing.

7.2.3 System calls tracing

7.2.4 Network sniffing CONTENTS

7.2.5 Sysinternals.

7.2.6 Valgrind

7.2.7 Emulators

7.3 Other tools.

7.3.1 Calculators

7.4 Do You Think Something Is Missing Here?

8 Case studies

8.1 Task manager practical joke (Windows Vista).

8.1.1 Using LEA to load values.

8.2 Color Lines game practical joke

8.3 Minesweeper (Windows XP)

8.3.1 Finding grid automatically.

8.3.2 Exercises.

8.4 Hacking Windows clock.

8.5 Dongles.

8.5.1 Example #1: MacOS Classic and PowerPC

8.5.2 Example #2: SCO OpenServer.

8.5.3 Example #3: MS-DOS

8.6 “QR9”: Rubik’s cube inspired amateur crypto-algorithm.

8.7 Encrypted database case #1

8.7.1 Base64 and entropy.

8.7.2 Is data compressed?

8.7.3 Is data encrypted?.

8.7.4 CryptoPP.

8.7.5 Cipher Feedback mode.

8.7.6 Initializing Vector

8.7.7 Structure of the buffer

8.7.8 Noise at the end.

8.7.9 Conclusion.

8.7.10 Post Scriptum: brute-forcing IV

8.8 Overclocking Cointerra Bitcoin miner.

8.9 Breaking simple executable cryptor.

8.9.1 Other ideas to consider

8.10 SAP.

8.10.1 About SAP client network traffic compression

8.10.2 SAP 6.0 password checking functions

8.11 Oracle RDBMS.

8.11.1V$VERSIONtable in the Oracle RDBMS

8.11.2X$KSMLRUtable in Oracle RDBMS

8.11.3V$TIMERtable in Oracle RDBMS

8.12 Handwritten assembly code.

8.12.1 EICAR test file

8.13 Demos.

8.13.1 10 PRINT CHR$(205.5+RND(1)); : GOTO

8.13.2 Mandelbrot set

8.14 Other examples.

9 Examples of reversing proprietary file formats

9.1 Primitive XOR-encryption.

9.1.1 Simplest ever XOR encryption

9.1.2 Norton Guide: simplest possible 1-byte XOR encryption

9.1.3 Simplest possible 4-byte XOR encryption

9.1.4 Simple encryption using XOR mask.

9.1.5 Simple encryption using XOR mask, case II.

9.2 Information entropy

9.2.1 Analyzing entropy in Mathematica

9.2.2 Conclusion.

9.2.3 Tools.

9.2.4 A word about primitive encryption like XORing

9.2.5 More about entropy of executable code.

9.2.6 PRNG

9.2.7 More examples. CONTENTS

9.2.8 Entropy of various files.

9.2.9 Making lower level of entropy

9.3 Millenium game save file.

9.4fortuneprogram indexing file.

9.4.1 Hacking

9.4.2 The files

9.5 Oracle RDBMS: .SYM-files

9.6 Oracle RDBMS: .MSB-files

9.6.1 Summary

9.7 Exercises.

9.8 Further reading.

10 Dynamic binary instrumentation

10.1 Using PIN DBI for XOR interception.

10.2 Cracking Minesweeper with PIN.

10.2.1 Intercepting all rand() calls

10.2.2 Replacing rand() calls with our function

10.2.3 Peeking into placement of mines

10.2.4 Exercise

10.3 Why “instrumentation”?

11 Other things

11.1 Executable files patching.

11.1.1 Text strings

11.1.2 x86 code.

11.2 Function arguments number statistics

11.3 Compiler intrinsic.

11.4 Compiler’s anomalies.

11.4.1 Oracle RDBMS 11.2 and Intel C++ 10.1.

11.4.2 MSVC 6.0..

11.4.3 Summary..

11.5 Itanium.

11.6 8086 memory model.

11.7 Basic blocks reordering..

11.7.1 Profile-guided optimization.

11.8 My experience with Hex-Rays 2.2.0.

11.8.1 Bugs..

11.8.2 Odd peculiarities..

11.8.3 Silence.

11.8.4 Comma..

11.8.5 Data types..

11.8.6 Long and messed expressions.

11.8.7 My plan..

11.8.8 Summary..

12 Books/blogs worth reading

12.1 Books and other materials..

12.1.1 Reverse Engineering.

12.1.2 Windows.

12.1.3 C/C++.

12.1.4 x86 / x86-64..

12.1.5 ARM.

12.1.6 Assembly language.

12.1.7 Java.

12.1.8 UNIX..

12.1.9 Programming in general.

12.1.10 Cryptography.

12.1.11 Dedication.

13 Communities

Afterword

13.1 Questions?.

Appendix CONTENTS

.1 x86.

.1.1 Terminology..

.1.2 General purpose registers..

.1.3 FPU registers..

.1.4 SIMD registers.

.1.5 Debugging registers.

.1.6 Instructions..

.1.7 npad.

.2 ARM..

.2.1 Terminology..

.2.2 Versions.

.2.3 32-bit ARM (AArch32).

.2.4 64-bit ARM (AArch64).

.2.5 Instructions..

.3 MIPS..

.3.1 Registers..

.3.2 Instructions..

.4 Some GCC library functions..

.5 Some MSVC library functions..

.6 Cheatsheets..

.6.1 IDA.

.6.2 OllyDbg.

.6.3 MSVC..

.6.4 GCC..

.6.5 GDB..

Acronyms used

Glossary

Index

Assembly Language for Beginners

Get our desktop app

Company

Features

Documentation

Resources