HTTP and Working

with the Web

The Hypertext Transfer Protocol (HTTP) is probably the most widely-used
application layer protocol. It was originally developed to allow academics to
share HTML documents. Nowadays, it is used as the core protocol of innumerable
applications across the Internet, and it is the principle protocol of the World
Wide Web.

In this chapter, we will cover the following topics:

• The HTTP protocol structure


• Using Python for talking to services through HTTP


• Downloading files


• HTTP capabilities, such as compression and cookies


• Handling errors


• URLs


• The Python standard library urllib package


• Kenneth Reitz's third-party Requests package

The urllib package is the recommended Python standard library package
for HTTP tasks. The standard library also has a low-level module called http.
Although this offers access to almost all aspects of the protocol, it has not been
designed for everyday use. The urllib package has a simpler interface, and it
deals with everything that we are going to cover in this chapter.