| Welcome to WebmasterWorld Guest from 184.108.40.206 |
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
|Become a Pro Member|
| 2:16 am on Jul 4, 2003 (gmt 0)|
After looking at my site root I did not see a robots.txt file. What exactly does this file do and how do you make one? What do you need to put into one and do you need to have one?
| 2:23 am on Jul 4, 2003 (gmt 0)|
robots.txt is used to forbid search engine crawlers access to certain files & directories on your site.
For instance if you didn't want to have your visitor stats crawled by the search engines, perhaps you would forbid the directory stats/ or where ever your stats page lives.
try [robotstxt.org...] or maybe
| 3:14 am on Jul 4, 2003 (gmt 0)|
>robots.txt is used to forbid search engine crawlers access to certain files & directories on your site.
This is assuming they follow robots.txt protocols. Don't rely on robots.txt file alone to protect sensitive information of temporary folders/files.
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved