| Welcome to WebmasterWorld Guest from 18.104.22.168 |
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
|Become a Pro Member|
|Only Allowing Googlebot, Msnbot and Slurp|
is it possible?
| 3:11 pm on Mar 11, 2005 (gmt 0)|
I need to reduce robothits as much as possible, so I would like to ban all robots except for googlebot, msnbot and slurp.
I didn't find a solution in previous threads, so help is greatly appreciated.
| 3:39 pm on Mar 11, 2005 (gmt 0)|
You just block everything by default, and then override with the three specific bots you want to allow. Try:
Of course, there are quite a few spiders which don't obey robots.txt, so you'll have to block those with mod_rewrite.
| 3:51 pm on Mar 11, 2005 (gmt 0)|
Just one more question:
is it okay to write:
| 4:28 pm on Mar 11, 2005 (gmt 0)|
globay, try the following:
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved