homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Weird 404 error hitting on robot.txt
anyone experienced this before?

 6:45 am on Nov 8, 2004 (gmt 0)

hello, my site is getting a lot of error404 hits on a page url with /robot.txt, where i actually didn't use a robot.txt in my site at all!--anyone experienced this and can tell me whats happening? and does this will affects my SE ranking?



 7:53 am on Nov 8, 2004 (gmt 0)

Hello redleaf

it is called a robots.txt (s)

Your log file will show a 404 because the search engine spider has requested the robots.txt file but it was unavailable, you don't have to have the file in your root directory but by having even a blank file will stop the errors from showing in the log file.

I personally use the robots.txt file, I think it is better not to give a spider a 404 but to serve the file, as for affecting SE ranking it won't affect the results but as an overall seo package it's important not to serve 404's.

Robots.txt Tutorial


 9:30 am on Nov 8, 2004 (gmt 0)

hey, thanks for the reply, heard it clear and loud.
will go thru the thread u provided now.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved