| Welcome to WebmasterWorld Guest from 18.104.22.168 |
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
|Become a Pro Member|
| HTTP Page robots.txt not found 404 error|
Could anyone tell me why this error generated on log files?
robots.txt is a standard way that search engines determine areas of your site to search.
for more see..
and wellcome. Search engine spiders are trying to find this file first. It tells them what they can and cannot spider. If it does not exist they think that everything on the website is spider food.
With robots.txt file the webmaster can ask spiders not to go to given directories or given files. Most robots follow the instructions but not all.
thanks all for your help!
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved