Reverse Engineering for Beginners

CHAPTER 19. MANIPULATING SPECIFIC BIT(S) CHAPTER 19. MANIPULATING SPECIFIC BIT(S)

The Linux 2.6 kernel is compiled with-mregparm=3option^34.

What this means to us is that the first 3 arguments are to be passed via registersEAX,EDXandECX, and the rest via the
stack. Of course, if the number of arguments is less than 3, only part of registers set is to be used.

So, let’s download Linux Kernel 2.6.31, compile it in Ubuntu:make vmlinux, open it inIDA, and find thedo_filp_open()
function. At the beginning, we see (the comments are mine):

Listing 19.6: do_filp_open() (linux kernel 2.6.31)

do_filp_open proc near

push ebp

mov ebp, esp

push edi

push esi

push ebx

mov ebx, ecx

add ebx, 1

sub esp, 98h

mov esi, [ebp+arg_4] ; acc_mode (5th arg)

test bl, 3

mov [ebp+var_80], eax ; dfd (1th arg)

mov [ebp+var_7C], edx ; pathname (2th arg)

mov [ebp+var_78], ecx ; open_flag (3th arg)

jnz short loc_C01EF684

mov ebx, ecx ; ebx <- open_flag

GCC saves the values of the first 3 arguments in the local stack. If that wasn’t done, the compiler would not touch these
registers, and that would be too tight environment for the compiler’sregister allocator.

Let’s find this fragment of code:

Listing 19.7: do_filp_open() (linux kernel 2.6.31)

loc_C01EF6B4: ; CODE XREF: do_filp_open+4F
test bl, 40h ; O_CREAT
jnz loc_C01EF810
mov edi, ebx
shr edi, 11h
xor edi, 1
and edi, 1
test ebx, 10000h
jz short loc_C01EF6D3
or edi, 2

0x40—is what theO_CREATmacro equals to.open_flaggets checked for the presence of the0x40bit, and if this bit is
1, the nextJNZinstruction is triggered.

19.1.2 ARM.

TheO_CREATbit is checked differently in Linux kernel 3.8.0.

Listing 19.8: linux kernel 3.8.0

struct file do_filp_open(int dfd, struct filename pathname,
const struct open_flags *op)
{
...
filp = path_openat(dfd, pathname, &nd, op, flags | LOOKUP_RCU);
...
}

static struct file path_openat(int dfd, struct filename pathname,
struct nameidata nd, const struct open_flags op, int flags)
{
...
error = do_last(nd, &path, file, op, &opened, pathname);
...

(^3) kernelnewbies.org/Linux_2_6_20#head-042c62f290834eb1fe0a1942bbf5bb9a4accbc8f
(^4) See alsoarch/x86/include/asm/calling.hfile in kernel tree