- On Windows, it spawns a new interpreter by using Windows-specific process cre-
ation tools, passing the pickled Process object in to the new process over a pipe,
and starting a “python -c” command line in the new process, which runs a special
Python-coded function in this package that reads and unpickles the Process and
invokes its run method.
We met pickling briefly in Chapter 1, and we will study it further later in this book.
The implementation is a bit more complex than this, and is prone to change over time,
of course, but it’s really quite an amazing trick. While the portable API generally hides
these details from your code, its basic structure can still have subtle impacts on the way
you’re allowed to use it. For instance:
- On Windows, the main process’s logic should generally be nested under a name
== main test as done here when using this module, so it can be imported freely
by a new interpreter without side effects. As we’ll learn in more detail in Chap-
ter 17, unpickling classes and functions requires an import of their enclosing mod-
ule, and this is the root of this requirement. - Moreover, when globals are accessed in child processes on Windows, their values
may not be the same as that in the parent at start time, because their module will
be imported into a new process. - Also on Windows, all arguments to Process must be pickleable. Because this in-
cludes target, targets should be simple functions so they can be pickled; they can-
not be bound or unbound object methods and cannot be functions created with a
lambda. See pickle in Python’s library manual for more on pickleability rules;
nearly every object type works, but callables like functions and classes must be
importable—they are pickled by name only, and later imported to recreate byte-
code. On Windows, objects with system state, such as connected sockets, won’t
generally work as arguments to a process target either, because they are not
pickleable. - Similarly, instances of custom Process subclasses must be pickleable on Windows
as well. This includes all their attribute values. Objects available in this package
(e.g., Lock in Example 5-29) are pickleable, and so may be used as both Process
constructor arguments and subclass attributes. - IPC objects in this package that appear in later examples like Pipe and Queue accept
only pickleable objects, because of their implementation (more on this in the next
section). - On Unix, although a child process can make use of a shared global item created in
the parent, it’s better to pass the object as an argument to the child process’s con-
structor, both for portability to Windows and to avoid potential problems if such
objects were garbage collected in the parent.
There are additional rules documented in the library manual. In general, though, if you
stick to passing in shared objects to processes and using the synchronization and
The multiprocessing Module| 247