Saylor URL: http://www.saylor.org/books Saylor.org
For example, if there is both an HTML and a PDF (portable document format) version of the same
content, the wise Webmaster will instruct search engine bots to index only one of the two to avoid being
penalized for duplicate content.
The robots.txt file is publicly accessible, so although it does not show restricted content, it can give an idea
of the content that a Web site owner wants to keep private. A robots.txt file needs to be created for each
subdomain.
Here is a robots.txt file with additional information.
Figure 13.5
Instructions to search engine robots can also be given in the meta tags. This means that instructions can
still be given if you only have access to the meta tags and not to the robots.txt file.
Make Sure It’s Not Broken
Make sure that both visitors to your Web site and search engines can see it all by following these
guidelines:
- Check for broken links—anything that you click that gives an error should be considered broken
and in need of fixing. - Validate your HTML and CSS in accordance withWorld Wide Web Consortium (W3C) guidelines.