Chapter 12We've created a Pool object with four separate processes and assigned this Pool
object to the workers variable. We've then mapped a function, analysis, to an
iterable queue of work to be done, using the pool of processes. Each process in the
workers pool will be assigned items from the iterable queue. In this case, the queue
is the result of the glob.glob(pattern) attribute, which is a sequence of file names.
As the analysis() function returns a result, the parent process that created the Pool
object can collect those results. This allows us to create several concurrently-built
Counter objects and merge them into a single, composite result.
If we start p processes in the pool, our overall application will include p+1 processes.
There will be one parent process and p children. This often works out well because
the parent process will have little to do after the subprocess pools are started.
Generally, the workers will be assigned to separate CPUs (or cores) and the parent
will share a CPU with one of the children in the Pool object.
The ordinary Linux parent/child process rules apply to the subprocesses
created by this module. If the parent crashes without properly collecting
final status from the child processes, then "zombie" processes can be left
running. For this reason, a process Pool object is a context manager.
When we use a pool via the with statement, at the end of the context, the
children are properly terminated.By default, a Pool object will have a number of workers based on the value of
the multiprocessing.cpu_count() function. This number is often optimal, and
simply using the with multiprocessing.Pool() as workers: attribute might
be sufficient.
In some cases, it can help to have more workers than CPUs. This might be true when
each worker has I/O-intensive processing. Having many worker processes waiting
for I/O to complete can improve the elapsed running time of an application.
If a given Pool object has p workers, this mapping can cut the processing time to
almost^1 p of the time required to process all of the logs serially. Pragmatically, there
is some overhead involved with communication between the parent and child
processes in the Pool object. Therefore, a four-core processor might only cut the
processing time in half.
The multiprocessing Pool object has four map-like methods to allocate work to a
pool: map(), imap(), imap_unordered(), and starmap(). Each of these is a variation
on the common theme of mapping a function to a pool of processes. They differ in
the details of allocating work and collecting results.