Welcome to WebmasterWorld Guest from 54.163.84.187

Forum Moderators: goodroi

Robots.txt

I better get this right...

   
5:53 pm on Feb 9, 2003 (gmt 0)

10+ Year Member



I want to add the robots.txt file to my site, but this is something that I really want to get right.

What would the basic file look like without excluding anything.

TIA,
Javi

6:30 pm on Feb 9, 2003 (gmt 0)

WebmasterWorld Senior Member heini is a WebmasterWorld Top Contributor of All Time 10+ Year Member



>basic
User-agent: * 
Disallow:

A great and very easy to understand tutorial is here:
[searchengineworld.com...]

And welcome to the board!

7:57 pm on Feb 9, 2003 (gmt 0)

10+ Year Member



Thanks for the link Heini. :)

Let's see if I undertand,

User-agent: *
Disallow: /forum/

Would just keep the spiders off my forum right? Everything else would be found.

Javi :)

8:16 pm on Feb 9, 2003 (gmt 0)

WebmasterWorld Senior Member heini is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Yup, that looks correct to me.
Everything else is allowed to be retrieved by any visitor, be it bot or browser.
4:16 am on Feb 11, 2003 (gmt 0)

10+ Year Member



my question is,

where does the file go?

in the root directory? of the WWW root?

my dir structure is....

/
www cgi-bin images
[all html content]

should i put robots.txt here with the HTML content? or one level up?

thanks

4:19 am on Feb 11, 2003 (gmt 0)

10+ Year Member



It goes in the same directory as your index or default file. Most people would call that the www root directory, I think.

Test with: [mydomain.com...]

 

Featured Threads

Hot Threads This Week

Hot Threads This Month