Forum Moderators: open

Message Too Old, No Replies

Is it possible to block all search engine robots?

         

Jesse_Smith

12:30 am on Apr 23, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I'll be doing PPC stuff, and I don't want those pages geting listed in the search engines. Is it possible to block all search engine robots with .htaccess?

marcs

12:49 am on Apr 23, 2003 (gmt 0)

10+ Year Member



Use a robots.txt file to block all robots:

User-agent: *
Disallow: /

marcs

12:51 am on Apr 23, 2003 (gmt 0)

10+ Year Member



To quick on the 'Submit' button :)

Replace

Disallow: /

with a Disallow entry for each of those page unless you do want to declare the entire site off limits to robots.

jdMorgan

2:33 am on Apr 23, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



robots.txt will only block "good" robots, those which obey robots.txt, but not the malicious or badly-coded 'bots.

A user-agent blocking rule and a spider trap script can take care of the bad ones; try this A Close to perfect .htaccess ban list [webmasterworld.com] post and this modified "bad-bot" script [webmasterworld.com] post for more info on how to take care of them.

HTH,
Jim