wind up in a deadlock state, both blocked on input calls waiting for events that never
occur.
Technically, by default stdout is just line-buffered when connected to a terminal, but
it is fully buffered when connected to other devices such as files, sockets, and the pipes
used here. This is why you see a script’s printed text in a shell window immediately as
it is produced, but not until the process exits or its buffer fills when its output stream
is connected to something else.
This output buffering is really a function of the system libraries used to access pipes,
not of the pipes themselves (pipes do queue up output data, but they never hide it from
readers!). In fact, it appears to occur in this example only because we copy the pipe’s
information over to sys.stdout, a built-in file object that uses stream buffering by de-
fault. However, such anomalies can also occur when using other cross-process tools.
In general terms, if your programs engage in a two-way dialog like this, there are a
variety of ways to avoid buffering-related deadlock problems:
- Flushes: As demonstrated in Examples 5-22 and 5-23, manually flushing output
pipe streams by calling the file object flush method is an easy way to force buffers
to be cleared. Use sys.stdout.flush for the output stream used by print. - Arguments: As introduced earlier in this chapter, the -u Python command-line flag
turns off full buffering for the sys.stdout stream in Python programs. Setting your
PYTHONUNBUFFERED environment variable to a nonempty value is equivalent to pass-
ing this flag but applies to every program run. - Open modes: It’s possible to use pipes themselves in unbuffered mode. Either use
low-level os module calls to read and write pipe descriptors directly, or pass a buffer
size argument of 0 (for unbuffered) or 1 (for line-buffered) to os.fdopen to disable
buffering in the file object used to wrap the descriptor. You can use open arguments
the same way to control buffering for output to fifo files (described in the next
section). Note that in Python 3.X, fully unbuffered mode is allowed only for binary
mode files, not text. - Command pipes: As mentioned earlier in this chapter, you can similarly specify
buffering mode arguments for command-line pipes when they are created by
os.popen and subprocess.Popen, but this pertains to the caller’s end of the pipe, not
those of the spawned program. Hence it cannot prevent delayed outputs from the
latter, but can be used for text sent to another program’s input pipe. - Sockets: As we’ll see later, the socket.makefile call accepts a similar buffering mode
argument for sockets (described later in this chapter and book), but in Python 3.X
this call requires buffering for text-mode access and appears to not support line-
buffered mode (more on this on Chapter 12). - Tools: For more complex tasks, we can also use higher-level tools that essentially
fool a program into believing it is connected to a terminal. These address programs
232 | Chapter 5: Parallel System Tools