The Multiprocessing and Threading Modules
Using the threading and queue modules
The Python threading package involves a number of constructs helpful for
building imperative applications. This module is not focused on writing functional
applications. We can make use of thread-safe queues in the queue module to pass
objects from thread to thread.
The threading module doesn't have a simple way to distribute work to various
threads. The API isn't ideally suited to functional programming.
As with the more primitive features of the multiprocessing module, we can try
to conceal the stateful and imperative nature of locks and queues. It seems easier,
however, to make use of the ThreadPoolExecutor method in the concurrent.
futures module. The ProcessPoolExecutor.map() method provides us with a
very pleasant interface to concurrent processing of the elements of a collection.
The use of the map() function primitive to allocate work seems to fit nicely with
our functional programming expectations. For this reason, it's best to focus on
the concurrent.futures module as the most accessible way to write concurrent
functional programs.
Designing concurrent processing
From a functional programming perspective, we've seen three ways to use
the map() function concept applied to data items concurrently. We can use any
one of the following:
- multiprocessing.Pool
- concurrent.futures.ProcessPoolExecutor
- concurrent.futures.ThreadPoolExecutor
These are almost identical in the way we interact with them; all three have a map()
method that applies a function to items of an iterable collection. This fits elegantly
with other functional programming techniques. The performance is different because
of the nature of concurrent threads versus concurrent processes.
As we stepped through the design, our log analysis application decomposed into
two overall areas:
- The lower-level parsing: This is generic parsing that will be used by almost
any log analysis application - The higher-level analysis application: This is more specific filtering and
reduction focused on our application needs