homepage Welcome to WebmasterWorld Guest from 54.226.166.224
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Weird 404 error hitting on robot.txt
anyone experienced this before?
redleaf




msg:1525798
 6:45 am on Nov 8, 2004 (gmt 0)

hello, my site is getting a lot of error404 hits on a page url with /robot.txt, where i actually didn't use a robot.txt in my site at all!--anyone experienced this and can tell me whats happening? and does this will affects my SE ranking?

 

ncw164x




msg:1525799
 7:53 am on Nov 8, 2004 (gmt 0)

Hello redleaf

it is called a robots.txt (s)

Your log file will show a 404 because the search engine spider has requested the robots.txt file but it was unavailable, you don't have to have the file in your root directory but by having even a blank file will stop the errors from showing in the log file.

I personally use the robots.txt file, I think it is better not to give a spider a 404 but to serve the file, as for affecting SE ranking it won't affect the results but as an overall seo package it's important not to serve 404's.

Robots.txt Tutorial
[searchengineworld.com...]

redleaf




msg:1525800
 9:30 am on Nov 8, 2004 (gmt 0)

ncw164x,
hey, thanks for the reply, heard it clear and loud.
will go thru the thread u provided now.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved