Can someone help me write a robots.txt?

[budbiss]

I want to allow all spiders...should be pretty simple, right?

Also, if you go to www.mysite.com...you are redirected to www.mysite.com/catalog/...so should the robots.txt go under www.mysite.com or www.mysite.com/catalog/?

[Terabytes]

you can find examples of robots.txt within the root of almost (almost) any domain...

an example would be the M$ website... simply type www.microsoft.com/robots.txt

they are all very similar throughout most domains, however you can learn quite a bit from some...an example would be here at webmasterworld.com quite a few disallowed robots within their robots.txt

the file itself is place in the root of the domain (IE same place your index.html, etc is placed)

hope that helps

[pmac]

Here is a real good robots.txt tutorial [searchengineworld.com].

[JohnieWalker]

Here's the official one.

robotstxt.org [robotstxt.org]

[plonkeroo]

Here's a simple example:-

User-agent: *
Disallow: /cgi-bin/
Disallow: /whateverdirectory/
Disallow: /anotherdictory/
Disallow: /yet anotherdirectory/email.html

The wildcard "*" means that all bots are welcome.