I'm not understanding the different makes of bots - Sitemaps, Meta Data, and robots.txt forum at WebmasterWorld - WebmasterWorld

Forum Moderators: goodroi

Message Too Old, No Replies

I'm not understanding the different makes of bots

Can I stop some, while allowing others?

tomequin

9:42 pm on Jul 1, 2004 (gmt 0)

10+ Year Member

Hi. Is there a way I can stop Google from crawling a site and still allow other bots (Yahoo, MSN) to crawl?
Thanks.

encyclo

10:11 pm on Jul 1, 2004 (gmt 0)

WebmasterWorld Senior Member

10+ Year Member

Sure thing - just add this to your robots.txt file in your document root:

User-agent: Googlebot 
Disallow: /

That will stop Googlebot, but not the others.