HTTP and Working with the Web
Cookies are necessary because the server has no other way of tracking a client
between requests. HTTP is called a stateless protocol. It doesn't contain an explicit
mechanism for a server to know for sure that two requests have come from the
same client. Without cookies to allow the server to add some uniquely identifying
information to the requests, things such as shopping carts (which were the original
problem that cookies were developed to solve) would become impossible to build,
because the server would not be able to determine which basket goes with
which request.
We may need to handle cookies in Python because without them, some sites don't
behave as expected. When using Python, we may also want to access the parts
of a site which require a login, and the login sessions are usually maintained
through cookies.
Cookie handling
We're going to discuss how to handle cookies with urllib. First, we need to create a
place for storing the cookies that the server will send us:
from http.cookiejar import CookieJar
cookie_jar = CookieJar()
Next, we build something called an urllib opener. This will automatically extract
the cookies from the responses that we receive and then store them in our cookie jar:
from urllib.request import build_opener, HTTPCookieProcessor
opener = build_opener(HTTPCookieProcessor(cookie_jar))
Then, we can use our opener to make an HTTP request:
opener.open('http://www.github.com')
Lastly, we can check that the server has sent us some cookies:
len(cookie_jar)
2
Whenever we use opener to make further requests, the HTTPCookieProcessor
functionality will check our cookie_jar to see if it contains any cookies for that site
and then it will automatically add them to our requests. It will also add any further
cookies that are received to the cookie jar.
The http.cookiejar module also contains a FileCookieJar class, that works in the
same way as CookieJar, but it provides an additional function for easily saving the
cookies to a file. This allows persistence of cookies across Python sessions.