download task proceeds, the program as a whole shouldn’t stall—it still must respond
to screen redraws, mouse clicks, and so on.
Parallel processing comes to the rescue here, too. By performing such long-running
tasks in parallel with the rest of the program, the system at large can remain responsive
no matter how busy some of its parts may be. Moreover, the parallel processing model
is a natural fit for structuring such programs and others; some tasks are more easily
conceptualized and coded as components running as independent, parallel entities.
There are two fundamental ways to get tasks running at the same time in Python—
process forks and spawned threads. Functionally, both rely on underlying operating
system services to run bits of Python code in parallel. Procedurally, they are very dif-
ferent in terms of interface, portability, and communication. For instance, at this writ-
ing direct process forks are not supported on Windows under standard Python (though
they are under Cygwin Python on Windows).
By contrast, Python’s thread support works on all major platforms. Moreover, the
os.spawn family of calls provides additional ways to launch programs in a platform-
neutral way that is similar to forks, and the os.popen and os.system calls and
subprocess module we studied in Chapters 2 and 3 can be used to portably spawn
programs with shell commands. The newer multiprocessing module offers additional
ways to run processes portably in many contexts.
In this chapter, which is a continuation of our look at system interfaces available to
Python programmers, we explore Python’s built-in tools for starting tasks in parallel,
as well as communicating with those tasks. In some sense, we’ve already begun doing
so—os.system, os.popen, and subprocess, which we learned and applied over the last
three chapters, are a fairly portable way to spawn and speak with command-line pro-
grams, too. We won’t repeat full coverage of those tools here.
Instead, our emphasis in this chapter is on introducing more direct techniques—forks,
threads, pipes, signals, sockets, and other launching techniques—and on using Py-
thon’s built-in tools that support them, such as the os.fork call and the threading,
queue, and multiprocessing modules. In the next chapter (and in the remainder of this
book), we use these techniques in more realistic programs, so be sure you understand
the basics here before flipping ahead.
One note up front: although the process, thread, and IPC mechanisms we will explore
in this chapter are the primary parallel processing tools in Python scripts, the third party
domain offers additional options which may serve more advanced or specialized roles.
As just one example, the MPI for Python system allows Python scripts to also employ
the Message Passing Interface (MPI) standard, allowing Python programs to exploit
multiple processors in various ways (see the Web for details). While such specific ex-
tensions are beyond our scope in this book, the fundamentals of multiprocessing that
we will explore here should apply to more advanced techniques you may encounter in
your parallel futures.
178 | Chapter 5: Parallel System Tools