Welcome to WebmasterWorld Guest from 54.160.163.163

Forum Moderators: goodroi

Message Too Old, No Replies

how to block all robots except for 2?

possible?

     

ionchannels

3:32 pm on Jun 28, 2006 (gmt 0)

10+ Year Member



I am just getting hammered by so many useless bots. I want to block all bots except for googlebot and slurp. Is there anyway to do this with robots.txt?
Thanks,
Christian

cheesehead2

8:15 pm on Jun 28, 2006 (gmt 0)

5+ Year Member



This should work:

#block all
User-agent: *
Disallow: /
#except
User-agent: Slurp
Disallow:
User-agent: msnbot
Disallow:
User-agent: Googlebot
Disallow:

jimbeetle

9:00 pm on Jun 28, 2006 (gmt 0)

WebmasterWorld Senior Member jimbeetle is a WebmasterWorld Top Contributor of All Time 10+ Year Member



You have that backwards. All bots, including Slurp, msnbot and Googlebot will read and obey the first Disallow.

Try this:

User-agent: Slurp
User-agent: Googlebot
Disallow:

User-agent: *
Disallow: /

Slurp and Google bot will go on their merry ways when they see the first directive, happily gobbling down pages. Others will see the second and be off to some other bloke's site.

larryhatch

9:11 pm on Jun 28, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I never understood this. All I see are 'disallows'.
First 2 robots specifically, then a wild-card for the rest.
Why doesn't this disallow all of them? -Larry

kevinpate

9:16 pm on Jun 28, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



What you're missing is the effect of what
comes after the :

Disallow:
(the above line says disallow nothing on site.)

Disallow: /
(the above line says disallow everything on site.)

larryhatch

9:17 am on Jun 29, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Thanks KevinP: That makes perfect sense and explains everything. -Larry

ionchannels

12:30 pm on Jun 29, 2006 (gmt 0)

10+ Year Member



Thanks guys, I will try that. It seems so simple, but I am always afraid to do something which will discourage googlebot. I didn't realize that the SE bots read robots.txt that way (sequentially I mean). Very cool, thanks again.
 

Featured Threads

Hot Threads This Week

Hot Threads This Month