Welcome to WebmasterWorld Guest from 54.145.53.251

Forum Moderators: goodroi

Message Too Old, No Replies

Confirm I've done this right please

     
4:29 pm on Feb 19, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:June 23, 2003
posts:48
votes: 0


The contents of my robots.txt file are:

User-agent: googlebot
Disallow: *
User-agent: scooter
Disallow: *
User-agent: lycos
Disallow: *

I'm trying to ban google, altavista and lycos.

Thanks

4:33 pm on Feb 19, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 22, 2003
posts:1022
votes: 0


Hiya,

Is it not

Disallow: /

/ rather than the *?
The * might work am open to be corrected on this one!

-gs

4:35 pm on Feb 19, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:June 23, 2003
posts:48
votes: 0


You're correct according to [robotstxt.org...]

Thanks. The bit I'm unsure about is if i can do

User-Agent: this, that, google, lycos, another
Disallow: /

4:37 pm on Feb 19, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 22, 2003
posts:1022
votes: 0


Don't think so just checked WW one and some other larger websites and they list each one on an individual basis...

edit according to link you gave you could use the * for all known robots i.e.
User-agent: *
Disallow

This would ban all known robots that obey the robots.txt

-gs

4:43 pm on Feb 19, 2004 (gmt 0)

Administrator

WebmasterWorld Administrator bakedjake is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Apr 8, 2003
posts:3783
votes: 2


AF, According to A Standard for Robot Exclusion [robotstxt.org], you are correct.

It should be:

User-agent: googlebot
Disallow: /
User-agent: scooter
Disallow: /
User-agent: lycos
Disallow: /

5:04 pm on Feb 19, 2004 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member pageoneresults is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Apr 27, 2001
posts:12166
votes: 51


It really should be...

User-agent: googlebot 
Disallow: /

User-agent: scooter 
Disallow: /

User-agent: lycos 
Disallow: /
5:04 pm on Feb 19, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:June 23, 2003
posts:48
votes: 0


"This would ban all known robots that obey the robots.txt"

Is it possible to ban everything except Examplebot?

Does this work?

Allow: Examplebot

5:07 pm on Feb 19, 2004 (gmt 0)

Administrator

WebmasterWorld Administrator bakedjake is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Apr 8, 2003
posts:3783
votes: 2


por: that's what i meant. ;-)

Does this work?

It depends if Examplebot honors the allow directive. robots.txt, don't forget, is not access control. It's a voluntary thing that the robots do.

Not all spiders read robots.txt, and some spiders accept proprietary parameters in robots.txt.

5:11 pm on Feb 19, 2004 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member pageoneresults is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Apr 27, 2001
posts:12166
votes: 51


Here's a great topic from jdMorgan in regards to the robots.txt file...

Put your robots.txt on a diet [webmasterworld.com]