homepage Welcome to WebmasterWorld Guest from 54.167.96.124
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
[404] HTTP Page robots.txt not found 404 error
Strike




msg:1527625
 12:11 pm on Dec 5, 2003 (gmt 0)

Could anyone tell me why this error generated on log files?

Cheers!

 

davidpbrown




msg:1527626
 12:34 pm on Dec 5, 2003 (gmt 0)

robots.txt is a standard way that search engines determine areas of your site to search.

for more see..
[searchengineworld.com ]

risto




msg:1527627
 12:37 pm on Dec 5, 2003 (gmt 0)

Hello Strike,

and wellcome. Search engine spiders are trying to find this file first. It tells them what they can and cannot spider. If it does not exist they think that everything on the website is spider food.

With robots.txt file the webmaster can ask spiders not to go to given directories or given files. Most robots follow the instructions but not all.

Strike




msg:1527628
 12:44 pm on Dec 10, 2003 (gmt 0)

thanks all for your help!

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved