Open Directory Project 153
when consensus has been reached.
In July 1998, the directory became multilingual with the addition of the World top-level category. The remainder of
the directory lists only English language sites. By May 2005, seventy-five languages were represented. The growth
rate of the non-English components of the directory has been greater than the English component since 2002. While
the English component of the directory held almost 75% of the sites in 2003, the World level grew to over 1.5
million sites as of May 2005, forming roughly one-third of the directory. The ontology in non-English categories
generally mirrors that of the English directory, although exceptions which reflect language differences are quite
common.
Several of the top-level categories have unique characteristics. The Adult category is not present on the directory
homepage but it is fully available in the RDF dump that ODP provides. While the bulk of the directory is categorized
primarily by topic, the Regional category is categorized primarily by region. This has led many to view ODP as two
parallel directories: Regional and Topical.
On November 14, 2000, a special directory within the Open Directory was created for people under 18 years of
age.[17] Key factors distinguishing this "Kids and Teens" area from the main directory are:- stricter guidelines which limit the listing of sites to those which are targeted or "appropriate" for people under 18
years of age;[18] - category names as well as site descriptions use vocabulary which is "age appropriate";
- age tags on each listing distinguish content appropriate for kids (age 12 and under), teens (13 to 15 years old) and
mature teens (16 to 18 years old); - • Kids and Teens content is available as a separate RDF dump;
- • editing permissions are such that the community is parallel to that of the Open Directory.
By May 2005, this portion of the Open Directory included over 32,000 site listings.
Since early 2004, the whole site has been in UTF-8 encoding. Prior to this, the encoding used to be ISO 8859-1 for
English language categories and a language-dependent character set for other languages. The RDF dumps have been
encoded in UTF-8 since early 2000.
Maintenance
Directory listings are maintained by editors. While some editors focus on the addition of new listings, others focus
on maintaining the existing listings. This includes tasks such as the editing of individual listings to correct spelling
and/or grammatical errors, as well as monitoring the status of linked sites. Still others go through site submissions to
remove spam and duplicate submissions.
Robozilla is a Web crawler written to check the status of all sites listed in ODP. Periodically, Robozilla will flag
sites which appear to have moved or disappeared and editors follow up to check the sites and take action. This
process is critical for the directory in striving to achieve one of its founding goals: to reduce the link rot in web
directories. Shortly after each run, the sites marked with errors are automatically moved to the unreviewed queue
where editors may investigate them when time permits.
Due to the popularity of the Open Directory and its resulting impact on search engine rankings (See PageRank),
domains with lapsed registration that are listed on ODP have attracted domain hijacking, an issue that has been
addressed by regularly removing expired domains from the directory.
While corporate funding and staff for the ODP have diminished in recent years, volunteers have created editing tools
such as linkcheckers to supplement Robozilla, category crawlers, spellcheckers, search tools that directly sift a recent
RDF dump, bookmarklets to help automate some editing functions, mozilla based add-ons,[19] and tools to help work
through unreviewed queues.