Google gives 404 error on robots.txt

Googlebot not able to find robots.txt

[Govind Uppaluri]

I have just started a site.
I have created a robots.txt file and placed it in the web root directory.
I am able to access the file from web browser (http://www.domainname/robots.txt)

Most validators I could find on the internet recognize and validate this file,
Only Google is gving 404 error.

From Google sitemaps also I get the same error.
Inktomi Slurp and few other bots have read the file successfully.

Please help. I am all out of ideas.

[pageoneresults]

Hello Govind, Welcome to WebmasterWorld!

I have created a robots.txt file and placed it in the web root directory. I am able to access the file from web browser (http://www.domainname/robots.txt)

Did you mean http://www.example.com/robots.txt

Note the .com, I just want to verify that.

Have you checked the headers being returned by your robots.txt file? You can do a quick check by clicking on the Control Panel link at top left (next to the WebmasterWorld logo. On the left under Plugins, third link, Server Headers. Enter the direct path to your robots.txt file which should be...

http://www.example.com/robots.txt

The header response returned should be a 200 OK. If it is returning a 404 Not Found, then of course something is wrong. First we want to make sure that it is returning a 404 as stated.

P.S. If you can browse to your robots.txt file, that means it is working. Do you have some sort of custom 404 in place?

[Govind Uppaluri]

Thank you. That was fast!

I did nothing, yet now the file validates from Google sitemaps.
Apparantly, Google sitemaps gets the robots.txt file once a day.
FYI, I did check with the .com included. ie, http://www.example.com

Here is the quote from Google, once the file was validated.
' If you have made changes to this file since we downloaded it, the text we show may be different than your latest version. We check for a new robots.txt file approximately once per day. '

Still the question is how did this come about?
I was fiddling with file permissions yesterday. You think that could be it?

Appreciate it. Thanx a lot.

[pageoneresults]

Is it possible your site was down at the time those errors were being reported? That would surely return a 404 for your robots.txt file. ;)

[kulinar]

I have the same problem. On google sitemaps, section robots.txt google says:

Your robots URL......http://mydomain.com/robots.txt
Last downloaded......Not Found
Status...............Not Found

We didn't find a robots.txt file on your site and so on.

I have three sitemaps submitted (3 different sites) and have the same problem with other sites. That is strange because all my robots.txt files exist. Should I hardcore them in sitemaps file?

[masterchief]

Same here for two of my sites

Your robots URL:
Last downloaded: Not Found
Status: Not Found

[kulinar]

Probably our servers were down during google robots.txt download. ;)