Forum Moderators: open

Message Too Old, No Replies

googlebot question

         

jeyval

5:06 am on May 31, 2003 (gmt 0)

10+ Year Member



Hi, is there a reason of why googlebot sometimes start with a crawler10, others 11, etc?

For instance, in a particular site which I suspect to be banned it starts:

crawler10.googlebot.com - - [29/May/2003:06:33:26 -0400] "GET /robots.txt HTTP/1.0" 404 204 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"

crawler10.googlebot.com - - [29/May/2003:06:33:26 -0400] "GET / HTTP/1.0" 200 50017 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"

Is anything wrong with the way it starts?

Thanks

conor

3:38 pm on May 31, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



It could be any of their crawlers that hits your site first. Nothing to worry about.

jeyval

3:51 pm on May 31, 2003 (gmt 0)

10+ Year Member



Thanks Connor - I just got the answer to this from other means as well and apparently it shows the initial 404 only because I dont have a robots.txt file but them immediately after it gives the 200 as okay

As you say, nothing to worry about it

What I cannot understand them if googlebot keeps spidering me fine everyday why my site has totally dessapeared from Google index and google directory and the pr bar is grey

Just does not make any sense and its driving me totally insane

Thanks very much :)