communication tools provided by this package, your code will usually be portable and
correct. Let’s look next at a few of those tools in action.
IPC Tools: Pipes, Shared Memory, and Queues
While the processes created by this package can always communicate using general
system-wide tools like the sockets and fifo files we met earlier, the multiprocessing
module also provides portable message passing tools specifically geared to this purpose
for the processes it spawns:
- Its Pipe object provides an anonymous pipe, which serves as a connection between
two processes. When called, Pipe returns two Connection objects that represent the
ends of the pipe. Pipes are bidirectional by default, and allow arbitrary pickleable
Python objects to be sent and received. On Unix they are implemented internally
today with either a connected socket pair or the os.pipe call we met earlier, and
on Windows with named pipes specific to that platform. Much like the Process
object described earlier, though, the Pipe object’s portable API spares callers from
such things. - Its Value and Array objects implement shared process/thread-safe memory for
communication between processes. These calls return scalar and array objects
based in the ctypes module and created in shared memory, with access synchron-
ized by default. - Its Queue object serves as a FIFO list of Python objects, which allows multiple pro-
ducers and consumers. A queue is essentially a pipe with extra locking mechanisms
to coordinate more arbitrary accesses, and inherits the pickleability constraints of
Pipe.
Because these devices are safe to use across multiple processes, they can often serve to
synchronize points of communication and obviate lower-level tools like locks, much
the same as the thread queues we met earlier. As usual, a pipe (or a pair of them) may
be used to implement a request/reply model. Queues support more flexible models; in
fact, a GUI that wishes to avoid the limitations of the GIL might use the
multiprocessing module’s Process and Queue to spawn long-running tasks that post
results, rather than threads. As mentioned, although this may incur extra start-up
overhead on some platforms, unlike threads today, tasks coded this way can be as truly
parallel as the underlying platform allows.
One constraint worth noting here: this package’s pipes (and by proxy, queues) pickle
the objects passed through them, so that they can be reconstructed in the receiving
process (as we’ve seen, on Windows the receiver process may be a fully independent
Python interpreter). Because of that, they do not support unpickleable objects; as sug-
gested earlier, this includes some callables like bound methods and lambda functions
(see file multi-badq.py in the book examples package for a demonstration of code that
violates this constraint). Objects with system state, such as sockets, may fail as well.
248 | Chapter 5: Parallel System Tools