Web Design

336

Prevent Pages from Being Listed on Search Engines ...

3 Type Disallow:.

4 Enter either an asterisk, a
path to a directory in the
site, or a specific filename.

1 Create a new blank document
in your editor.

2 Type User-agent: *.

Prevent Pages from Being Listed on Search Engines

O

ccasionally, you may have pages within your site that you do not want to be indexed on search

engines. For example, you may have test pages on your site that show developmental stages of

upcoming features, or possibly pages you have set up for your own personal use to which you do not

want to grant others easy access. A robots.txt file can prevent engines from indexing certain pages

by including one or more Disallow statements. An asterisk disallows indexing of the entire site;

a specific folder disallows indexing that folder, and a particular page blocks indexing of the page.

1

2

3 4