Refer to Python’s library manual for more information on the pickler; it supports ad-
ditional interfaces that classes may use to customize its behavior, which we’ll bypass
here in the interest of space. Also check out marshal, a module that serializes an object
too, but can handle only simple object types. pickle is more general than marshal and
is normally preferred.
An additional related module, _pickle, is a C-coded optimization of pickle, and is
automatically used by pickle internally if available; it need not be selected or used
directly. The shelve module inherits this optimization automatically by proxy. I haven’t
explained shelve yet, but I will now.
Shelve Files
Pickling allows you to store arbitrary objects on files and file-like objects, but it’s still
a fairly unstructured medium; it doesn’t directly support easy access to members of
collections of pickled objects. Higher-level structures can be added to pickling, but they
are not inherent:
- You can sometimes craft your own higher-level pickle file organizations with the
underlying filesystem (e.g., you can store each pickled object in a file whose name
uniquely identifies the object), but such an organization is not part of pickling itself
and must be manually managed. - You can also store arbitrarily large dictionaries in a pickled file and index them by
key after they are loaded back into memory, but this will load and store the entire
dictionary all at once when unpickled and pickled, not just the entry you are in-
terested in.
Shelves provide structure for collections of pickled objects that removes some of these
constraints. They are a type of file that stores arbitrary Python objects by key for later
retrieval, and they are a standard part of the Python system. Really, they are not much
of a new topic—shelves are simply a combination of the DBM files and object pickling
we just met:
- To store an in-memory object by key, the shelve module first serializes the object
to a string with the pickle module, and then it stores that string in a DBM file by
key with the dbm module. - To fetch an object back by key, the shelve module first loads the object’s serialized
string by key from a DBM file with the dbm module, and then converts it back to
the original in-memory object with the pickle module.
Because shelve uses pickle internally, it can store any object that pickle can: strings,
numbers, lists, dictionaries, cyclic objects, class instances, and more. Because shelve
uses dbm internally, it inherits all of that module’s capabilities, as well as its portability
constraints.
Shelve Files | 1315