Functional Python Programming

Chapter 12

The lower-level parsing can be decomposed into four stages:

• Reading all the lines from multiple source logfiles. This was the
  local_gzip() mapping from file name to a sequence of lines.


• Creating simple namedtuples from the lines of log entries in a collection of
  files. This was the access_iter() mapping from text lines to Access objects.


• Parsing the details of more complex fields such as dates and URLs.
  This was the access_detail_iter() mapping from Access objects to
  AccessDetails objects.


• Rejecting uninteresting paths from the logs. We can also think of this
  as passing only the interesting paths. This was more of a filter than
  a map operation. This was a collection of filters bundled into the
  path_filter() function.

We defined an overall analysis() function that parsed and analyzed a given
logfile. It applied the higher-level filter and reduction to the results of the lower-level
parsing. It can also work with a wild-card collection of files.

Given the number of mappings involved, we can see several ways to decompose
this problem into work that can be mapped to into a pool of threads or processes.
Here are some of the mappings we can consider as design alternatives:

• Map the analysis() function to individual files. We use this as a consistent
  example throughout this chapter.


• Refactor the local_gzip() function out of the overall analysis() function.
  We can now map the revised analysis() function to the results of the
  local_gzip() function.


• Refactor the access_iter(local_gzip(pattern)) function out of the
  overall analysis() function. We can map this revised analysis() function
  against the iterable sequence of the Access objects.


• Refactor the access_detail_iter(access-iter(local_gzip(pattern)))
  function into a separate iterable. We will then map the path_filter()
  function and the higher-level filter and reduction against the iterable
  sequence of the AccessDetail objects.


• We can also refactor the lower-level parsing into a function that is separate
  from the higher-level analysis. We can map the analysis filter and reduction
  against the output from the lower-level parsing.

All of these are relatively simple restructurings of the example application. The
benefit of using functional programming techniques is that each part of the overall
process can be defined as a mapping. This makes it practical to consider different
architectures to locate an optimal design.