homepage Welcome to WebmasterWorld Guest from 54.145.172.149
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Only Allowing Googlebot, Msnbot and Slurp
is it possible?
globay

10+ Year Member



 
Msg#: 571 posted 3:11 pm on Mar 11, 2005 (gmt 0)

I need to reduce robothits as much as possible, so I would like to ban all robots except for googlebot, msnbot and slurp.

I didn't find a solution in previous threads, so help is greatly appreciated.

 

encyclo

WebmasterWorld Senior Member encyclo us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 571 posted 3:39 pm on Mar 11, 2005 (gmt 0)

You just block everything by default, and then override with the three specific bots you want to allow. Try:

User-agent: *
Disallow: /

User-agent: Googlebot
Disallow:

User-agent: MSNBot
Disallow:

User-agent: Slurp
Disallow:

Of course, there are quite a few spiders which don't obey robots.txt, so you'll have to block those with mod_rewrite.

globay

10+ Year Member



 
Msg#: 571 posted 3:51 pm on Mar 11, 2005 (gmt 0)

Thank you!

Just one more question:

is it okay to write:

User-agent: *
Disallow: /

User-agent: Googlebot
Disallow:
Disallow: /subfolder/

ThomasB

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 571 posted 4:28 pm on Mar 11, 2005 (gmt 0)

globay, try the following:

User-agent: *
Disallow: /

User-agent: Googlebot
Disallow: /subfolder/

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved