Confirm I've done this right please

[Essel]

The contents of my robots.txt file are:

User-agent: googlebot
Disallow: *
User-agent: scooter
Disallow: *
User-agent: lycos
Disallow: *

I'm trying to ban google, altavista and lycos.

Thanks

[Alternative Future]

Hiya,

Is it not

Disallow: /

/ rather than the *?
The * might work am open to be corrected on this one!

-gs

[Essel]

You're correct according to [robotstxt.org...]

Thanks. The bit I'm unsure about is if i can do

User-Agent: this, that, google, lycos, another
Disallow: /

[Alternative Future]

Don't think so just checked WW one and some other larger websites and they list each one on an individual basis...

edit according to link you gave you could use the * for all known robots i.e.
User-agent: *
Disallow

This would ban all known robots that obey the robots.txt

-gs

[bakedjake]

AF, According to A Standard for Robot Exclusion [robotstxt.org], you are correct.

It should be:

User-agent: googlebot
Disallow: /
User-agent: scooter
Disallow: /
User-agent: lycos
Disallow: /

[pageoneresults]

It really should be...

User-agent: googlebot 
Disallow: /

User-agent: scooter 
Disallow: /

User-agent: lycos 
Disallow: /

[Essel]

"This would ban all known robots that obey the robots.txt"

Is it possible to ban everything except Examplebot?

Does this work?

Allow: Examplebot

[bakedjake]

por: that's what i meant. ;-)

Does this work?

It depends if Examplebot honors the allow directive. robots.txt, don't forget, is not access control. It's a voluntary thing that the robots do.

Not all spiders read robots.txt, and some spiders accept proprietary parameters in robots.txt.

[pageoneresults]

Here's a great topic from jdMorgan in regards to the robots.txt file...

Put your robots.txt on a diet [webmasterworld.com]