Welcome to WebmasterWorld Guest from 220.127.116.11
Can anyone explain the following server log entry?
18.104.22.168 - - [27/Feb/2005:16:12:11 -0500] "GET /robots.txt HTTP/1.0" 302 216 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)"
22.214.171.124 - - [27/Feb/2005:16:12:12 -0500] "GET /robots.txt HTTP/1.0" 200 66 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)"
126.96.36.199 - - [27/Feb/2005:16:12:12 -0500] "GET / HTTP/1.0" 302 206 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)"
188.8.131.52 - - [27/Feb/2005:16:12:12 -0500] "GET / HTTP/1.0" 200 13443 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)"
I am trying to prevent 302 redirects from occuring as I am under the impression that Google interpretes them as possible spamming!
Is the fact that I am running my site on a virtual server the reason that I get a 302 followed by the 200? I am NOT trying to fool googlebot by doing any sort of redirection.
This has only happened in the last couple of hours and I don't know if googlebot has returned to spider other pages. If it does it might happen when I am not watching - being the end of the month I won't be able to see the activity in the top 20 browser visits.
There is nothing wrong with 302, 301 and other redirects.
What is strange is that sometimes the robot gets a response of 200 straight away, other times it gets the temporarily moved code followed by the 200.
As I was saying I was wondering if the fact that my server is a "virtul" one? It was an inexpensive solution for hosting and I do not have full control over the server - for instance I can't edit the .htaccess file directly - even for security. To protect a directory I have a utility that writes to it although I can't see it!
I can't telnet to my server but ftp is no problem. My suspicion is that there is something strange going on. As I say I can't see the .htaccess file but I can see the other user's domains that are hosted on the same server! There is probably some redirecting to the same server as you suggest but there is nothing I can really do about that other than getting a proper hosting package or setting up my own server. (which I am thinking of doing).
Google did come back, but I have other suspicions that Google is being "put off" when it gets a 302 when it goes after my robots.txt file. If I had a META redirect on my base page that would look like a page highjack - especially if the wait time was set to zero. But there is no way that robots.txt can do this!
Anyway I was wondering if anyone else has seen this.
For example if your site directs all requests for "example.com" to "www.example.com" (or vice versa) then you'd see two requests - one for the "wrong" URL which gets redirected, and then another for the "right" URL which gets delivered.
As for GB being "put off" by a 301/302 on your robots.txt file - I doubt you're the only person who redirects all traffic from one version of the site to another so there's no point worrying about it.