Welcome to WebmasterWorld Guest from 54.157.7.205

Forum Moderators: goodroi

Message Too Old, No Replies

Only Allowing Googlebot, Msnbot and Slurp

is it possible?

     
3:11 pm on Mar 11, 2005 (gmt 0)

Full Member

10+ Year Member

joined:Jan 9, 2003
posts:330
votes: 0


I need to reduce robothits as much as possible, so I would like to ban all robots except for googlebot, msnbot and slurp.

I didn't find a solution in previous threads, so help is greatly appreciated.

3:39 pm on Mar 11, 2005 (gmt 0)

Senior Member from CA 

WebmasterWorld Senior Member encyclo is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Aug 31, 2003
posts:9073
votes: 4


You just block everything by default, and then override with the three specific bots you want to allow. Try:

User-agent: *
Disallow: /

User-agent: Googlebot
Disallow:

User-agent: MSNBot
Disallow:

User-agent: Slurp
Disallow:

Of course, there are quite a few spiders which don't obey robots.txt, so you'll have to block those with mod_rewrite.

3:51 pm on Mar 11, 2005 (gmt 0)

Full Member

10+ Year Member

joined:Jan 9, 2003
posts:330
votes: 0


Thank you!

Just one more question:

is it okay to write:

User-agent: *
Disallow: /

User-agent: Googlebot
Disallow:
Disallow: /subfolder/

4:28 pm on Mar 11, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Dec 5, 2002
posts:1562
votes: 0


globay, try the following:

User-agent: *
Disallow: /

User-agent: Googlebot
Disallow: /subfolder/