Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

Weird 404 error hitting on robot.txt

anyone experienced this before?



6:45 am on Nov 8, 2004 (gmt 0)

10+ Year Member

hello, my site is getting a lot of error404 hits on a page url with /robot.txt, where i actually didn't use a robot.txt in my site at all!--anyone experienced this and can tell me whats happening? and does this will affects my SE ranking?


7:53 am on Nov 8, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

Hello redleaf

it is called a robots.txt (s)

Your log file will show a 404 because the search engine spider has requested the robots.txt file but it was unavailable, you don't have to have the file in your root directory but by having even a blank file will stop the errors from showing in the log file.

I personally use the robots.txt file, I think it is better not to give a spider a 404 but to serve the file, as for affecting SE ranking it won't affect the results but as an overall seo package it's important not to serve 404's.

Robots.txt Tutorial


9:30 am on Nov 8, 2004 (gmt 0)

10+ Year Member

hey, thanks for the reply, heard it clear and loud.
will go thru the thread u provided now.

Featured Threads

Hot Threads This Week

Hot Threads This Month