The Multiprocessing and Threading Modules
The resulting Counter objects from the analysis() function are combined into a
single resulting Counter object. This aggregate gives us an overall summary of a
number of logfiles. This processing is not any faster than the previous example.
The use of the map_async() function allows the parent process to do additional
work while waiting for the children to finish.
More complex multiprocessing architectures
The multiprocessing package supports a wide variety of architectures. We can
easily create multiprocessing structures that span multiple servers and provide
formal authentication techniques to create a necessary level of security. We can
pass objects from process to process using queues and pipes. We can share memory
between processes. We can also share lower-level locks between processes as a way
to synchronize access to shared resources such as files.
Most of these architectures involve explicitly managing state among several working
processes. Using locks and shared memory, in particular, are imperative in nature
and don't fit well with a functional programming approach.
We can, with some care, treat queues and pipes in a functional manner. Our
objective is to decompose a design into producer and consumer functions. A
producer can create objects and insert them into a queue. A consumer will take
objects out of a queue and process them, perhaps putting intermediate results into
another queue. This creates a network of concurrent processors and the workload is
distributed among these various processes. Using the pycsp package can simplify
the queue-based exchange of messages among processes. For more information, visit
https://pypi.python.org/pypi/pycsp.
This design technique has some advantages when designing a complex application
server. The various subprocesses can exist for the entire life of the server, handling
individual requests concurrently.
Using the concurrent.futures module
In addition to the multiprocessing package, we can also make use of the
concurrent.futures module. This also provides a way to map data to a concurrent
pool of threads or processes. The module API is relatively simple and similar in
many ways to the multiprocessing.Pool() function's interface.