Welcome to WebmasterWorld Guest from 107.22.7.35

Forum Moderators: goodroi

Message Too Old, No Replies

Confirm I've done this right please

     
4:29 pm on Feb 19, 2004 (gmt 0)

10+ Year Member



The contents of my robots.txt file are:

User-agent: googlebot
Disallow: *
User-agent: scooter
Disallow: *
User-agent: lycos
Disallow: *

I'm trying to ban google, altavista and lycos.

Thanks

4:33 pm on Feb 19, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Hiya,

Is it not

Disallow: /

/ rather than the *?
The * might work am open to be corrected on this one!

-gs

4:35 pm on Feb 19, 2004 (gmt 0)

10+ Year Member



You're correct according to [robotstxt.org...]

Thanks. The bit I'm unsure about is if i can do

User-Agent: this, that, google, lycos, another
Disallow: /

4:37 pm on Feb 19, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Don't think so just checked WW one and some other larger websites and they list each one on an individual basis...

edit according to link you gave you could use the * for all known robots i.e.
User-agent: *
Disallow

This would ban all known robots that obey the robots.txt

-gs

4:43 pm on Feb 19, 2004 (gmt 0)

WebmasterWorld Administrator bakedjake is a WebmasterWorld Top Contributor of All Time 10+ Year Member



AF, According to A Standard for Robot Exclusion [robotstxt.org], you are correct.

It should be:

User-agent: googlebot
Disallow: /
User-agent: scooter
Disallow: /
User-agent: lycos
Disallow: /

5:04 pm on Feb 19, 2004 (gmt 0)

WebmasterWorld Senior Member pageoneresults is a WebmasterWorld Top Contributor of All Time 10+ Year Member



It really should be...

User-agent: googlebot 
Disallow: /

User-agent: scooter 
Disallow: /

User-agent: lycos 
Disallow: /
5:04 pm on Feb 19, 2004 (gmt 0)

10+ Year Member



"This would ban all known robots that obey the robots.txt"

Is it possible to ban everything except Examplebot?

Does this work?

Allow: Examplebot

5:07 pm on Feb 19, 2004 (gmt 0)

WebmasterWorld Administrator bakedjake is a WebmasterWorld Top Contributor of All Time 10+ Year Member



por: that's what i meant. ;-)

Does this work?

It depends if Examplebot honors the allow directive. robots.txt, don't forget, is not access control. It's a voluntary thing that the robots do.

Not all spiders read robots.txt, and some spiders accept proprietary parameters in robots.txt.

5:11 pm on Feb 19, 2004 (gmt 0)

WebmasterWorld Senior Member pageoneresults is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Here's a great topic from jdMorgan in regards to the robots.txt file...

Put your robots.txt on a diet [webmasterworld.com]

 

Featured Threads

Hot Threads This Week

Hot Threads This Month