Expert C Programming

frighten an operating system.

The kernel development group at Sun had one obscure bug that was very difficult to track
down. The symptom was that kernel memory was getting overwritten at random,
occasionally panicking the system.

Two of our top engineers worked on this, and they noticed that it was always the 19th byte
from the beginning of a memory block that was being creamed. This is an uncommon
offset, unlike the more usual 2, 4, or 8 that occur everywhere. One of the engineers had a
brainwave and used the offset to home in on the bug. He suggested they use the kernel
debugger kadb to disassemble the kernel binary image (it took an hour!) into an ASCII file.
They then grepped the file, looking for "store" instructions where the operand indicated
offset 19! One of these was almost guaranteed to be the instruction causing the corruption.

There were only eight of these, and they were all in the subsystem that dealt with process
control. Now they were pretty sure where the problem was, and just needed to find out what
it was. Further effort eventually yielded the cause: a race condition on a process control
structure. This resulted in one thread marking the memory for return to the system before
another thread had truly finished with it. Result: the kernel memory allocator then gave the
memory away to someone else, but the process control code thought it still held it, and
wrote into it, causing the otherwise impossible-to-find corruption!

Debugging an OS kernel with grep—what a concept. Sometimes even source tools can help
solve runtime problems!

While on the subject of useful tools, Table 0-5 lists some ways to see exactly what the configuration
of a Sun system is. However, none of these tools can help you unless you practice using them.

Table 6-5. Tools to Help Identify Your Hardware

What It Identifies Typical Output How to Invoke It

Kernel architecture sun4c /usr/kvm/arch -k

Any OS patches

applied

no patches are installed /usr/bin/showrev -p

Various hard ware

things

lots /usr/sbin/prtconf

CPU clock rate 40MHz processor /usr/sbin/psrinfo -v

hostid 554176fe /usr/ucb/hostid

Memory 32Mb Displayed on power up

Serial number 4290302 Displayed on power up

ROM revision 2.4.1 Displayed on power up

Mounted disk 198Mb disk /usr/bin/df -F ufs -k

Swap space 40Mb /etc/swap -s

Ethernet address 8:0:20:f:8c:60 /usr/sbin/ifconfig -a

The ethernet address is built into the

machine