homepage Welcome to WebmasterWorld Guest from 54.242.18.190
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
I'm not understanding the different makes of bots
Can I stop some, while allowing others?
tomequin




msg:1528596
 9:42 pm on Jul 1, 2004 (gmt 0)

Hi. Is there a way I can stop Google from crawling a site and still allow other bots (Yahoo, MSN) to crawl?
Thanks.

 

encyclo




msg:1528597
 10:11 pm on Jul 1, 2004 (gmt 0)

Sure thing - just add this to your robots.txt file in your document root:

User-agent: Googlebot 
Disallow: /

That will stop Googlebot, but not the others.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved