SEO: Search Engine Optimization Bible

Y

ou’re nearing the end of the book, so you should have a pretty good

handle on what exactly robots, spiders, and crawlers are, right? No

doubt you do, but did you know that there is much more to these

Internet creatures than just the fact that they crawl from one web site to

another?

Spiders, robots, crawlers, or whatever else you choose to call them can deter-

mine how well you rank in search engines, so it’s best to make friends with

them as quickly as possible. Certain strategies will help you find favor with

these crawlers (which is the name we’ll use to lump them all together), and

others, unfortunately, will help you find your way right out of search engine

rankings.

A sad fact of crawlers is that sometimes you’ll be treated unfairly when you’ve

done absolutely nothing wrong. Some of the strategies that you may want to

use to improve the optimization of your site can look suspicious to crawlers.

They’ll see those strategies and automatically penalize you for trying to spam

the search engine, even when you’re not.

That’s what makes it so important for you to understand not only the intimate

details about how crawlers work, but also about what makes them happy and

what makes them move on without looking back.

227

IN THIS CHAPTER

What are robots, crawlers, and

spiders?

What’s the Robot Exclusion Standard? ..............................................................................

Robots meta tag

Inclusion with XML site mapping

Robots, Spiders,

and Crawlers

75002c16.qxd:Layout 1 11/7/07 9:55 AM Page 227