Welcome to WebmasterWorld Guest from 107.20.72.188

Forum Moderators: goodroi

Message Too Old, No Replies

I'm not understanding the different makes of bots

Can I stop some, while allowing others?

     
9:42 pm on Jul 1, 2004 (gmt 0)

New User

10+ Year Member

joined:Feb 6, 2004
posts:24
votes: 0


Hi. Is there a way I can stop Google from crawling a site and still allow other bots (Yahoo, MSN) to crawl?
Thanks.
10:11 pm on July 1, 2004 (gmt 0)

Senior Member from CA 

WebmasterWorld Senior Member encyclo is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Aug 31, 2003
posts:9068
votes: 4


Sure thing - just add this to your robots.txt file in your document root:

User-agent: Googlebot 
Disallow: /

That will stop Googlebot, but not the others.

 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members