and reacquire the GIL around long-running operations to let the Python language por-
tions of other Python threads run during the wait. Specifically, the long-running C
extension function should release the lock on entry and reacquire it on exit when re-
suming Python code.
Also note that even though the Python code in Python threads cannot truly overlap in
time due to the GIL synchronization, the C-coded portions of threads can. Any number
may be running in parallel, as long as they do work outside the scope of the Python
virtual machine. In fact, C threads may overlap both with other C threads and with
Python language threads run in the virtual machine. Because of this, splitting code off
to C libraries is one way that Python applications can still take advantage of multi-CPU
machines.
Still, it may often be easier to leverage such machines by simply writing Python pro-
grams that fork processes instead of starting threads. The complexity of process and
thread code is similar. For more on C extensions and their threading requirements, see
Chapter 20. In short, Python includes C language tools (including a pair of GIL man-
agement macros) that can be used to wrap long-running operations in C-coded exten-
sions and that allow other Python language threads to run in parallel.
A process-based alternative: multiprocessing (ahead)
By now, you should have a basic grasp of parallel processes and threads, and Python’s
tools that support them. Later in this chapter, we’ll revisit both ideas to study the
multiprocessing module—a standard library tool that seeks to combine the simplicity
and portability of threads with the benefits of processes, by implementing a threading-
like API that runs processes instead of threads. It seeks to address the portability issue
of processes, as well as the multiple-CPU limitations imposed in threads by the GIL,
but it cannot be used as a replacement for forking in some contexts, and it imposes
some constraints that threads do not, which stem from its process-based model (for
instance, mutable object state is not directly shared because objects are copied across
process boundaries, and unpickleable objects such as bound methods cannot be as
freely used).
Because the multiprocessing module also implements tools to simplify tasks such as
inter-process communication and exit status, though, let’s first get a handle on Python’s
support in those domains as well, and explore some more process and thread examples
along the way.
Program Exits
As we’ve seen, unlike C, there is no “main” function in Python. When we run a program,
we simply execute all of the code in the top-level file, from top to bottom (i.e., in the
filename we listed in the command line, clicked in a file explorer, and so on). Scripts
Program Exits | 213