| Welcome to WebmasterWorld Guest from 220.127.116.11 |
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
|Pubcon Platinum Sponsor 2014|
After looking at my site root I did not see a robots.txt file. What exactly does this file do and how do you make one? What do you need to put into one and do you need to have one?
robots.txt is used to forbid search engine crawlers access to certain files & directories on your site.
For instance if you didn't want to have your visitor stats crawled by the search engines, perhaps you would forbid the directory stats/ or where ever your stats page lives.
try [robotstxt.org...] or maybe
>robots.txt is used to forbid search engine crawlers access to certain files & directories on your site.
This is assuming they follow robots.txt protocols. Don't rely on robots.txt file alone to protect sensitive information of temporary folders/files.
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved