Reversing and Interoperability
For a software engineer, interoperability can be a nightmare. From the indi-
vidual engineer’s perspective, interoperability means getting the software to
cooperate with software written by someone else. This other person can be
someone else working in the same company on the same product or the devel-
oper of some entirely separate piece of software. Modern software compo-
nents frequently interact: applications with operating systems, applications
with libraries, and applications with other applications.
Getting software to communicate with other components of the same pro-
gram, other programs, software libraries, and the operating system can be one
of the biggest challenges in large-scale software development. In many cases,
when you’re dealing with a third-party library, you have no access to the source
code of the component with which you’re interfacing. In such cases you’re
forced to rely exclusively on vendor-supplied documentation. Any seasoned
software developer knows that this rarely turns out to be a smooth and easy
process. The documentation almost always neglects to mention certain func-
tions, parameters, or entire features.
One excellent example is the Windows operating system, which has histori-
cally contained hundreds of such undocumented APIs. These APIs were kept
undocumented for a variety of reasons, such as to maintain compatibility with
other Windows platforms. In fact, many people have claimed that Windows
APIs were kept undocumented to give Microsoft an edge over one software
vendor or another. The Microsoft product could take advantage of a special
undocumented API to provide better features, which would not be available to
a competing software vendor.
This chapter teaches techniques for digging into any kind of third-party
code on your own. These techniques can be useful in a variety of situations, for
example when you have insufficient documentation (or no documentation at
all) or when you are experiencing problems with third-party code and you
have no choice but to try to solve these problems on your own. Sure, you
should only consider this approach of digging into other people’s code as a
last resort and at least try and get answers through the conventional channels.
Unfortunately, I’ve often found that going straight to the code is actually faster
than trying to contact some company’s customer support department when
you have a very urgent and very technicalquestion on your hands.Laying the Ground Rules
Before starting the first reversing session, let’s define some of the ground rules
for every reversing session in this book. First of all, the reversing sessions in142 Chapter 5