robots.txt broken link 404's

[nadsab]

Always been curious about this one, I don't have a robots.txt file in my site but it shows up often in my log files as a broken link. I think whenever googlebot comes. Does this mean that google is always looking for this and can't find it?

Should I even place one on site even if it's blank for google, does it cause problems if there is not a robots file there?

[skipfactor]

Does this mean that google is always looking for this and can't find it?

Yes

I have an old, old site that's never had robots.txt & hardly ever gets changed and deepbot & freshbot are regular visitors

[Oaf357]

Most search engines (good ones at least) look for the robots.txt file right off the bat. When they don't find one it will generate a 404 error (as seen in your error_log). Whether using one helps or not I don't know. I structure sites I build in a certain manner and there is always at least on directory I don't want spidered so I always use a robots.txt file.

[skipfactor]

I use it for protected directories but what I don't like is that anyone can read your robots.txt & see which folders are protected if you disallow them, I know, paranoia...

[jdMorgan]

nadsab,

If you'd like to stop seeing all those 404 errors, just add this robots.txt to your site's root directory:

User-agent: *
Disallow:

This is a valid robots.txt which disallows no files. So all it does is prevent you from getting 404 errors in your log file when robots come to crawl.

Include the trailing blank line - It was in the specification, and some primitive robots require it, or did in the past.

HTH,
Jim