302 Redirects and Google

Originally posted in the wrong place - Sorry I'm new!

[Tempufugit]

I currently use a virtual server to host my site. It runs Apache on a Sun Cobalt box and I don't have complete control over the behaviour of its server responses.

Can anyone explain the following server log entry?

66.249.64.66 - - [27/Feb/2005:16:12:11 -0500] "GET /robots.txt HTTP/1.0" 302 216 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)"
66.249.64.66 - - [27/Feb/2005:16:12:12 -0500] "GET /robots.txt HTTP/1.0" 200 66 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)"
66.249.64.66 - - [27/Feb/2005:16:12:12 -0500] "GET / HTTP/1.0" 302 206 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)"
66.249.64.66 - - [27/Feb/2005:16:12:12 -0500] "GET / HTTP/1.0" 200 13443 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)"

I am trying to prevent 302 redirects from occuring as I am under the impression that Google interpretes them as possible spamming!

Is the fact that I am running my site on a virtual server the reason that I get a 302 followed by the 200? I am NOT trying to fool googlebot by doing any sort of redirection.

This has only happened in the last couple of hours and I don't know if googlebot has returned to spider other pages. If it does it might happen when I am not watching - being the end of the month I won't be able to see the activity in the top 20 browser visits.

Confused
TF

[davidof]

Google asks for robots.txt, the server sends a 302 response code telling google that the resource has move temporarily, Google then goes and fetches the resource. I assume from the logs that you are just redirecting to the same server which seems non-optimal to me. Presumably you have two domains and redirect one to the other. In that case a 301 redirect would be more logical. Have you been fiddling with your .htaccess file?

There is nothing wrong with 302, 301 and other redirects.

[Tempufugit]

Thanks David,

What is strange is that sometimes the robot gets a response of 200 straight away, other times it gets the temporarily moved code followed by the 200.

As I was saying I was wondering if the fact that my server is a "virtul" one? It was an inexpensive solution for hosting and I do not have full control over the server - for instance I can't edit the .htaccess file directly - even for security. To protect a directory I have a utility that writes to it although I can't see it!

I can't telnet to my server but ftp is no problem. My suspicion is that there is something strange going on. As I say I can't see the .htaccess file but I can see the other user's domains that are hosted on the same server! There is probably some redirecting to the same server as you suggest but there is nothing I can really do about that other than getting a proper hosting package or setting up my own server. (which I am thinking of doing).

Google did come back, but I have other suspicions that Google is being "put off" when it gets a 302 when it goes after my robots.txt file. If I had a META redirect on my base page that would look like a page highjack - especially if the wait time was set to zero. But there is no way that robots.txt can do this!

Anyway I was wondering if anyone else has seen this.

TF

[Dreamquick]

Depends on what it's requested...

For example if your site directs all requests for "example.com" to "www.example.com" (or vice versa) then you'd see two requests - one for the "wrong" URL which gets redirected, and then another for the "right" URL which gets delivered.

As for GB being "put off" by a 301/302 on your robots.txt file - I doubt you're the only person who redirects all traffic from one version of the site to another so there's no point worrying about it.

- Tony