Welcome to WebmasterWorld Guest from 54.224.230.193

Forum Moderators: goodroi

Message Too Old, No Replies

how to block all robots except for 2?

possible?

     
3:32 pm on Jun 28, 2006 (gmt 0)

Junior Member

10+ Year Member

joined:Mar 22, 2005
posts:95
votes: 0


I am just getting hammered by so many useless bots. I want to block all bots except for googlebot and slurp. Is there anyway to do this with robots.txt?
Thanks,
Christian
8:15 pm on June 28, 2006 (gmt 0)

New User

10+ Year Member

joined:June 28, 2006
posts:4
votes: 0


This should work:

#block all
User-agent: *
Disallow: /
#except
User-agent: Slurp
Disallow:
User-agent: msnbot
Disallow:
User-agent: Googlebot
Disallow:
9:00 pm on June 28, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member jimbeetle is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Oct 26, 2002
posts:3295
votes: 6


You have that backwards. All bots, including Slurp, msnbot and Googlebot will read and obey the first Disallow.

Try this:

User-agent: Slurp
User-agent: Googlebot
Disallow:

User-agent: *
Disallow: /

Slurp and Google bot will go on their merry ways when they see the first directive, happily gobbling down pages. Others will see the second and be off to some other bloke's site.

9:11 pm on June 28, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 13, 2004
posts:1425
votes: 0


I never understood this. All I see are 'disallows'.
First 2 robots specifically, then a wild-card for the rest.
Why doesn't this disallow all of them? -Larry
9:16 pm on June 28, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Dec 2, 2002
posts:1167
votes: 0


What you're missing is the effect of what
comes after the :

Disallow:
(the above line says disallow nothing on site.)

Disallow: /
(the above line says disallow everything on site.)

9:17 am on June 29, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 13, 2004
posts:1425
votes: 0


Thanks KevinP: That makes perfect sense and explains everything. -Larry
12:30 pm on June 29, 2006 (gmt 0)

Junior Member

10+ Year Member

joined:Mar 22, 2005
posts:95
votes: 0


Thanks guys, I will try that. It seems so simple, but I am always afraid to do something which will discourage googlebot. I didn't realize that the SE bots read robots.txt that way (sequentially I mean). Very cool, thanks again.