Chapter 8 ■ CaChes and Message Queues
14 4
• A publisher-subscriber or fanout topology looks like a pipeline but with a key difference.
While the pipeline makes sure that every queued message is delivered to exactly one
consumer—since, after all, it would be wasteful for two thumbnail servers to be assigned the
same photograph—subscribers typically want to receive all of the messages that are being
queued the publishers. Alternatively, subscribers can specify a filter that narrows their interest
to messages with a particular format. This kind of queue can be used to power external
services that need to push events to the outside world. It can also form a fabric that a machine
room full of servers can use to advertise which systems are up, which are going down for
maintenance, and which can even publish the addresses of other message queues as they are
created and destroyed.• Finally, the request-reply pattern is the most complex because messages have to make a
round-trip. Both of the previous patterns placed very little responsibility on the producer of
a message: the producer connects to the queue and transmits its message, and it is done. But
a message queue client that makes a request has to stay connected and wait for the reply to
be delivered to it. The queue, to support this, has to feature some sort of addressing scheme
by which replies can be directed to the correct client, perhaps out of thousands of connected
clients, which is still sitting and waiting for it. But for all of its underlying complexity, this is
probably the most powerful pattern of all. It allows the load of dozens or hundreds of clients
to be spread equally across large numbers of servers without any effort beyond setting up the
message queue. Since a good message queue will allow servers to attach and detach without
losing messages, this topology also allows servers to be brought down for maintenance in a
way that is invisible to the population of client machines.Request-reply queues are a great way to connect lightweight workers that can run together by the hundreds
on a particular machine—say like the threads of a web server front end—to database clients or file servers that
sometimes need to be called in to do heavier work on the front end’s behalf. The request-reply pattern is a natural fit
for RPC mechanisms, with an added benefit not usually offered by simpler RPC systems; that is, many consumers or
producers can all be attached to the same queue in a fan-in or fan-out work pattern, without either group of clients
knowing the difference.
Using Message Queues from Python
The most popular message queues are implemented as stand-alone servers. All of the various tasks out of which you
choose to build your application—producers, consumers, filters, and RPC services—can then attach to the message
queue and not have to learn each other’s addresses or even identity. The AMQP protocol is one of the most widely
implemented language-agnostic message queue protocols, and it is supported by open source servers that you can
install such as RabbitMQ, the Apache Qpid server, and a number of other projects.
Many programmers never learn a messaging protocol themselves. Instead, they lean on third-party libraries that
package up the benefits of a message queue for easy consumption through an API. Many Python programmers who
use the Django web framework, for example, use the popular Celery distributed task queue instead of learning AMQP
themselves. A library can also offer protocol independence by supporting other back-end services. In Celery’s case,
you can use the simple Redis key-value store as your “message queue” instead of a dedicate messaging appliance.
However, for the purposes of this book, an example that does not require the installation of a full-fledged
separate message queue server is more convenient, so I will cover ØMQ, the Zero Message Queue, which was created
by the same company as AMQP but moves the messaging intelligence from a centralized broker into every one of
your message client programs. Embedding the ØMQ library in each of your programs, in other words, lets your code
spontaneously build a messaging fabric without the need for a centralized broker. This involves several differences
in approach from an architecture based on a central broker that can provide reliability, redundancy, retransmission,
and persistence to disk. A good summary of the advantages and disadvantages is provided at the ØMQ web site:
http://www.zeromq.org/docs:welcome-from-amqp.