we’ve already studied? Before we get into the details, I want to begin with a few words
about why you may (or may not) care about this package. In more specific terms,
although this module’s performance may not compete with that of pure threads or
process forks for some applications, this module offers a compelling solution for many:
- Compared to raw process forks, you gain cross-platform portability and powerful
IPC tools. - Compared to threads, you essentially trade some potential and platform-
dependent extra task start-up time for the ability to run tasks in truly parallel fash-
ion on multi-core or multi-CPU machines.
On the other hand, this module imposes some constraints and tradeoffs that threads
do not:
- Since objects are copied across process boundaries, shared mutable state does not
work as it does for threads—changes in one process are not generally noticed in
the other. Really, freely shared state may be the most compelling reason to use
threads; its absence in this module may prove limiting in some threading contexts. - Because this module requires pickleability for both its processes on Windows, as
well as some of its IPC tools in general, some coding paradigms are difficult or
nonportable—especially if they use bound methods or pass unpickleable objects
such as sockets to spawned processes.
For instance, common coding patterns with lambda that work for the threading module
cannot be used as process target callables in this module on Windows, because they
cannot be pickled. Similarly, because bound object methods are also not pickleable, a
threaded program may require a more indirect design if it either runs bound methods
in its threads or implements thread exit actions by posting arbitrary callables (possibly
including bound methods) on shared queues. The in-process model of threads supports
such direct lambda and bound method use, but the separate processes of
multiprocessing do not.
In fact we’ll write a thread manager for GUIs in Chapter 10 that relies on queueing
in-process callables this way to implement thread exit actions—the callables are queued
by worker threads, and fetched and dispatched by the main thread. Because the
threaded PyMailGUI program we’ll code in Chapter 14 both uses this manager to queue
bound methods for thread exit actions and runs bound methods as the main action of
a thread itself, it could not be directly translated to the separate process model implied
by multiprocessing.
Without getting into too many details here, to use multiprocessing, PyMailGUI’s ac-
tions might have to be coded as simple functions or complete process subclasses for
pickleability. Worse, they may have to be implemented as simpler action identifiers
dispatched in the main process, if they update either the GUI itself or object state in
general —pickling results in an object copy in the receiving process, not a reference to
the original, and forks on Unix essentially copy an entire process. Updating the state
244 | Chapter 5: Parallel System Tools