Welcome to WebmasterWorld Guest from 54.205.176.107

Forum Moderators: goodroi

Message Too Old, No Replies

Robots.txt

I better get this right...

     
5:53 pm on Feb 9, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Jan 5, 2003
posts:63
votes: 0


I want to add the robots.txt file to my site, but this is something that I really want to get right.

What would the basic file look like without excluding anything.

TIA,
Javi

6:30 pm on Feb 9, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member heini is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Jan 31, 2001
posts:4404
votes: 0


>basic
User-agent: * 
Disallow:

A great and very easy to understand tutorial is here:
[searchengineworld.com...]

And welcome to the board!

7:57 pm on Feb 9, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Jan 5, 2003
posts:63
votes: 0


Thanks for the link Heini. :)

Let's see if I undertand,

User-agent: *
Disallow: /forum/

Would just keep the spiders off my forum right? Everything else would be found.

Javi :)

8:16 pm on Feb 9, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member heini is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Jan 31, 2001
posts:4404
votes: 0


Yup, that looks correct to me.
Everything else is allowed to be retrieved by any visitor, be it bot or browser.
4:16 am on Feb 11, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Jan 5, 2003
posts:82
votes: 0


my question is,

where does the file go?

in the root directory? of the WWW root?

my dir structure is....

/
www cgi-bin images
[all html content]

should i put robots.txt here with the HTML content? or one level up?

thanks

4:19 am on Feb 11, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Apr 18, 2002
posts:126
votes: 0


It goes in the same directory as your index or default file. Most people would call that the www root directory, I think.

Test with: [mydomain.com...]