| Welcome to WebmasterWorld Guest from 220.127.116.11 |
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
|Pubcon Platinum Sponsor 2014|
I better get this right...
| 5:53 pm on Feb 9, 2003 (gmt 0)|
I want to add the robots.txt file to my site, but this is something that I really want to get right.
What would the basic file look like without excluding anything.
| 6:30 pm on Feb 9, 2003 (gmt 0)|
A great and very easy to understand tutorial is here:
And welcome to the board!
| 7:57 pm on Feb 9, 2003 (gmt 0)|
Thanks for the link Heini. :)
Let's see if I undertand,
Would just keep the spiders off my forum right? Everything else would be found.
| 8:16 pm on Feb 9, 2003 (gmt 0)|
Yup, that looks correct to me.
Everything else is allowed to be retrieved by any visitor, be it bot or browser.
| 4:16 am on Feb 11, 2003 (gmt 0)|
my question is,
where does the file go?
in the root directory? of the WWW root?
my dir structure is....
www cgi-bin images
[all html content]
should i put robots.txt here with the HTML content? or one level up?
| 4:19 am on Feb 11, 2003 (gmt 0)|
It goes in the same directory as your index or default file. Most people would call that the www root directory, I think.
Test with: [mydomain.com...]
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved