AWstats query

/robots.txt and 404 error

[gord298]

When examining AWstats HTTP Error codes for my websites I have noticed that under 404 Document Not Found top of the list on every site, and accounting for nearly all the errors, is /robots.txt

This has me confused. I thought that /robots.txt was used to instruct search engines NOT to spider a specific area of the website. Because there are no parts of my websites I wish to hide from search engines I never use /robots.txt

Why then is /robots.txt listed as a 404 error? Also does the fact that it is listed affect my standing with the search engines in any way?

Can someone please enlighten an ignorant soul, and if I'm doing something wrong please point me in the right direction.

[sem4u]

You are getting a 404 error because the spiders are looking for a robots.txt file when they access your site. Having no robots file means that an error is shown in your stats.

To get around this just upload a blank robots.txt file or one that allows all spiders.

Having no robots.txt file will not harm your rankings but adding one will reduce the number of 404 errors in your logs.

[gord298]

sem4u, thanks for your advice :)