Welcome to WebmasterWorld Guest from 23.21.38.201

Forum Moderators: goodroi

Weird 404 error hitting on robot.txt

anyone experienced this before?

   
6:45 am on Nov 8, 2004 (gmt 0)

10+ Year Member



hello, my site is getting a lot of error404 hits on a page url with /robot.txt, where i actually didn't use a robot.txt in my site at all!--anyone experienced this and can tell me whats happening? and does this will affects my SE ranking?
7:53 am on Nov 8, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Hello redleaf

it is called a robots.txt (s)

Your log file will show a 404 because the search engine spider has requested the robots.txt file but it was unavailable, you don't have to have the file in your root directory but by having even a blank file will stop the errors from showing in the log file.

I personally use the robots.txt file, I think it is better not to give a spider a 404 but to serve the file, as for affecting SE ranking it won't affect the results but as an overall seo package it's important not to serve 404's.

Robots.txt Tutorial
[searchengineworld.com...]

9:30 am on Nov 8, 2004 (gmt 0)

10+ Year Member



ncw164x,
hey, thanks for the reply, heard it clear and loud.
will go thru the thread u provided now.
 

Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month