Robots.txt

I better get this right...

[JVB_Mktg]

I want to add the robots.txt file to my site, but this is something that I really want to get right.

What would the basic file look like without excluding anything.

TIA,
Javi

[heini]

>basic

User-agent: * 
Disallow:

A great and very easy to understand tutorial is here:
[searchengineworld.com...]

And welcome to the board!

[JVB_Mktg]

Thanks for the link Heini. :)

Let's see if I undertand,

User-agent: *
Disallow: /forum/

Would just keep the spiders off my forum right? Everything else would be found.

Javi :)

[heini]

Yup, that looks correct to me.
Everything else is allowed to be retrieved by any visitor, be it bot or browser.

[neh2008]

my question is,

where does the file go?

in the root directory? of the WWW root?

my dir structure is....

/
www cgi-bin images
[all html content]

should i put robots.txt here with the HTML content? or one level up?

thanks

[Jack_Straw]

It goes in the same directory as your index or default file. Most people would call that the www root directory, I think.

Test with: [mydomain.com...]