Robots.txt 168
References
[ 1 ]http:/ / http://www. robotstxt. org/ orig. html#status
[ 2 ]http:/ / http://www. robotstxt. org/ norobots-rfc. txt
[ 3 ]http:/ / http://www. youtube. com/ watch?v=KBdEwpRQRD0#t=196s
[ 4 ]"How can I reduce the number of requests you make on my web site?" (http:/ / help. yahoo. com/ l/ us/ yahoo/ search/ webcrawler/ slurp-03.
html). Yahoo! Slurp.. Retrieved 2007-03-31.
[ 5 ]"MSNBot is crawling a site too frequently" (http:/ / search. msn. com/ docs/ siteowner.
aspx?t=SEARCH_WEBMASTER_FAQ_MSNBotIndexing. htm& FORM=WFDD#D). Troubleshoot issues with MSNBot and site crawling..
Retrieved 2007-02-08.
[ 6 ]"About Ask.com: Webmasters" (http:/ / about. ask. com/ en/ docs/ about/ webmasters. shtml#15)..
[ 7 ]"Webmaster Help Center - How do I block Googlebot?" (http:/ / http://www. google. com/ support/ webmasters/ bin/ answer. py?hl=en&
answer=156449& from=40364).. Retrieved 2007-11-20.
[ 8 ]"How do I prevent my site or certain subdirectories from being crawled? - Yahoo Search Help" (http:/ / help. yahoo. com/ l/ us/ yahoo/
search/ webcrawler/ slurp-02. html).. Retrieved 2007-11-20.
[ 9 ]"Google's Hidden Interpretation of Robots.txt" (http:/ / blog. semetrical. com/ googles-secret-approach-to-robots-txt/ ).. Retrieved
2010-11-15.
[ 10 ]"Robots Exclusion Protocol - joining together to provide better documentation" (http:/ / http://www. bing. com/ community/ site_blogs/ b/
webmaster/ archive/ 2008/ 06/ 03/ robots-exclusion-protocol-joining-together-to-provide-better-documentation. aspx).. Retrieved 2009-12-03.
[ 11 ]"Yahoo! Search Blog - Webmasters can now auto-discover with Sitemaps" (http:/ / ysearchblog. com/ 2007/ 04/ 11/
webmasters-can-now-auto-discover-with-sitemaps/ ).. Retrieved 2009-03-23.
[ 12 ]"Search engines and dynamic content issues" (http:/ / ghita. org/ search-engines-dynamic-content-issues. html). MSNbot issues with
robots.txt.. Retrieved 2007-04-01.External links
- http://www.robotstxt.org - The Web Robots Pages (http:/ / http://www. robotstxt. org/ )
- History of robots.txt (http:/ / http://www. antipope. org/ charlie/ blog-static/ 2009/ 06/
how_i_got_here_in_the_end_part_3. html) - (how Charles Stross prompted its invention; original comment (http:/
/ yro. slashdot. org/ comments. pl?sid=377285& cid=21554125) on Slashdot) - Block or remove pages using a robots.txt file - Google Webmaster Tools Help = Using the robots.txt analysis tool
(http:/ / http://www. google. com/ support/ webmasters/ bin/ answer. py?hl=en& answer=156449) - About Robots.txt at the Mediawiki website (http:/ / http://www. mediawiki. org/ wiki/ Robots. txt)
- List of Bad Bots (http:/ / http://www. kloth. net/ internet/ badbots. php) - rogue robots and spiders which ignore these
guidelines - Wikipedia's Robots.txt - an example (http:/ / en. wikipedia. org/ robots. txt)
- Robots.txt Generator + Tutorial (http:/ / http://www. mcanerin. com/ EN/ search-engine/ robots-txt. asp)
- Robots.txt Generator Tool (http:/ / http://www. howrank. com/ Robots. txt-Tool. php)
- Robots.txt is not a security measure (http:/ / http://www. diovo. com/ 2008/ 09/ robotstxt-is-not-a-security-measure/ )