I am trying to prevent 302 redirects from occuring as I am under the impression that Google interpretes them as possible spamming!
Is the fact that I am running my site on a virtual server the reason that I get a 302 followed by the 200? I am NOT trying to fool googlebot by doing any sort of redirection.
This has only happened in the last couple of hours and I don't know if googlebot has returned to spider other pages. If it does it might happen when I am not watching - being the end of the month I won't be able to see the activity in the top 20 browser visits.
Google asks for robots.txt, the server sends a 302 response code telling google that the resource has move temporarily, Google then goes and fetches the resource. I assume from the logs that you are just redirecting to the same server which seems non-optimal to me. Presumably you have two domains and redirect one to the other. In that case a 301 redirect would be more logical. Have you been fiddling with your .htaccess file?
There is nothing wrong with 302, 301 and other redirects.
What is strange is that sometimes the robot gets a response of 200 straight away, other times it gets the temporarily moved code followed by the 200.
As I was saying I was wondering if the fact that my server is a "virtul" one? It was an inexpensive solution for hosting and I do not have full control over the server - for instance I can't edit the .htaccess file directly - even for security. To protect a directory I have a utility that writes to it although I can't see it!
I can't telnet to my server but ftp is no problem. My suspicion is that there is something strange going on. As I say I can't see the .htaccess file but I can see the other user's domains that are hosted on the same server! There is probably some redirecting to the same server as you suggest but there is nothing I can really do about that other than getting a proper hosting package or setting up my own server. (which I am thinking of doing).
Google did come back, but I have other suspicions that Google is being "put off" when it gets a 302 when it goes after my robots.txt file. If I had a META redirect on my base page that would look like a page highjack - especially if the wait time was set to zero. But there is no way that robots.txt can do this!
Anyway I was wondering if anyone else has seen this.
For example if your site directs all requests for "example.com" to "www.example.com" (or vice versa) then you'd see two requests - one for the "wrong" URL which gets redirected, and then another for the "right" URL which gets delivered.
As for GB being "put off" by a 301/302 on your robots.txt file - I doubt you're the only person who redirects all traffic from one version of the site to another so there's no point worrying about it.