Does Google have the hiccups today?

2 files, robots, 2-3 more files, robots, and on and on and on...

[pendanticist]

Greetings,

Is this what a bot looks like when it has the hiccups?

64.68.86.59 - - [14/Dec/2002:16:25:07 -0800] "GET /About_Site.html HTTP/1.0" 200 5937 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.86.59 - - [14/Dec/2002:16:25:12 -0800] "GET /Legal_Research_Dictionaries-Law-Libraries.html HTTP/1.0" 200 4921 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.86.59 - - [14/Dec/2002:16:25:49 -0800] "GET /Hmmmmm.html HTTP/1.0" 200 6112 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.86.122 - - [14/Dec/2002:16:26:02 -0800] "GET /robots.txt HTTP/1.0" 200 130 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.86.122 - - [14/Dec/2002:16:26:02 -0800] "GET /Criminology_Corrections.html HTTP/1.0" 200 8025 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.86.122 - - [14/Dec/2002:16:26:30 -0800] "GET /robots.txt HTTP/1.0" 200 130 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.86.122 - - [14/Dec/2002:16:26:30 -0800] "GET /Psychology_Behavioral.html HTTP/1.0" 200 4694 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.86.54 - - [14/Dec/2002:16:26:53 -0800] "GET /robots.txt HTTP/1.0" 200 130 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.86.54 - - [14/Dec/2002:16:26:56 -0800] "GET /Aboriginal_Tribes-Councils_P-Z.html HTTP/1.0" 200 13301 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.86.59 - - [14/Dec/2002:16:27:24 -0800] "GET /Webmastery_Miscellaneous.html HTTP/1.0" 200 7947 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.86.122 - - [14/Dec/2002:16:27:29 -0800] "GET /Reference.html HTTP/1.0" 200 6749 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.86.123 - - [14/Dec/2002:16:27:40 -0800] "GET /robots.txt HTTP/1.0" 200 130 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.86.123 - - [14/Dec/2002:16:27:40 -0800] "GET /Terrorism.html HTTP/1.0" 200 6826 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.86.122 - - [14/Dec/2002:16:27:51 -0800] "GET /robots.txt HTTP/1.0" 200 130 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.86.122 - - [14/Dec/2002:16:27:51 -0800] "GET /Mathematics.html HTTP/1.0" 200 10606 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.86.123 - - [14/Dec/2002:16:28:18 -0800] "GET /robots.txt HTTP/1.0" 200 130 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.86.123 - - [14/Dec/2002:16:28:18 -0800] "GET /Marketing_Yourself_Resume.html HTTP/1.0" 200 3183 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.86.79 - - [14/Dec/2002:16:28:19 -0800] "GET /robots.txt HTTP/1.0" 200 130 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.86.79 - - [14/Dec/2002:16:28:20 -0800] "GET /Presidential.html HTTP/1.0" 200 8727 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.86.123 - - [14/Dec/2002:16:28:42 -0800] "GET /Health.html HTTP/1.0" 200 25036 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.86.54 - - [14/Dec/2002:16:28:47 -0800] "GET /robots.txt HTTP/1.0" 200 130 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.86.54 - - [14/Dec/2002:16:28:48 -0800] "GET /Archaeology.html HTTP/1.0" 200 13357 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.86.54 - - [14/Dec/2002:16:29:06 -0800] "GET /Environmental_F-P.html HTTP/1.0" 200 10860 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.86.54 - - [14/Dec/2002:16:29:11 -0800] "GET /Aboriginal_Fisheries.html HTTP/1.0" 200 6880 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.86.54 - - [14/Dec/2002:16:29:23 -0800] "GET /Aboriginal_Friendship.html HTTP/1.0" 200 3520 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.86.122 - - [14/Dec/2002:16:29:29 -0800] "GET /robots.txt HTTP/1.0" 200 130 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.86.122 - - [14/Dec/2002:16:29:30 -0800] "GET /Human_Rights_H-X.html HTTP/1.0" 200 10843 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.86.122 - - [14/Dec/2002:16:29:48 -0800] "GET /Listserves.html HTTP/1.0" 200 5569 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.86.123 - - [14/Dec/2002:16:30:13 -0800] "GET /robots.txt HTTP/1.0" 200 130 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.86.123 - - [14/Dec/2002:16:30:13 -0800] "GET /Paleontology.html HTTP/1.0" 200 5767 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.86.79 - - [14/Dec/2002:16:30:14 -0800] "GET /robots.txt HTTP/1.0" 200 130 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.86.79 - - [14/Dec/2002:16:30:16 -0800] "GET /Webmastery.html HTTP/1.0" 200 9072 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.86.54 - - [14/Dec/2002:16:30:23 -0800] "GET /robots.txt HTTP/1.0" 200 130 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.86.59 - - [14/Dec/2002:16:30:23 -0800] "GET /robots.txt HTTP/1.0" 200 130 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.86.54 - - [14/Dec/2002:16:30:23 -0800] "GET /About_Search_Simple.html HTTP/1.0" 200 1610 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.86.59 - - [14/Dec/2002:16:30:29 -0800] "GET /Countries.html HTTP/1.0" 200 23364 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"

That list goes on, and on, and on, and on...

Any ideas what might be happening here?

Thanks.

Pendanticist.

[mack]

by the fact that you posted this , am I assuming correctly that these pages do not exist on your site?

[pendanticist]

by the fact that you posted this , am I assuming correctly that these pages do not exist on your site?

Uh, no you are not. These are indeed my own access log files.

Pendanticist.

[SuzyUK]

I have seen this too...only it wasn't a googlebot... I think it was FAST...but it kept repeating the call for robots.txt in between valid pages.

I didn't know either, but it has only happened once

Suzy

[pendanticist]

Hi Suzy,

I didn't know either, but it has only happened once

So, I suppose that eliminates anything causal on my part then.

Pendanticist.

[Lisa]

I see lots of Robots.txt file requests as well. And I questioned it when I saw it.

64.68.86.123 - - [13/Dec/2002:18:20:10 -0800] "GET /robots.txt HTTP/1.0" 200 101 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.86.54 - - [13/Dec/2002:18:38:39 -0800] "GET /robots.txt HTTP/1.0" 200 101 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.86.54 - - [13/Dec/2002:18:41:34 -0800] "GET /robots.txt HTTP/1.0" 200 101 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.86.123 - - [13/Dec/2002:18:43:04 -0800] "GET /robots.txt HTTP/1.0" 200 101 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.86.123 - - [13/Dec/2002:18:44:01 -0800] "GET /robots.txt HTTP/1.0" 200 101 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.86.122 - - [13/Dec/2002:18:44:40 -0800] "GET /robots.txt HTTP/1.0" 200 101 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.86.123 - - [13/Dec/2002:18:45:21 -0800] "GET /robots.txt HTTP/1.0" 200 101 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.86.59 - - [13/Dec/2002:18:54:33 -0800] "GET /robots.txt HTTP/1.0" 200 101 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.86.79 - - [13/Dec/2002:18:56:02 -0800] "GET /robots.txt HTTP/1.0" 200 101 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.86.122 - - [13/Dec/2002:18:56:41 -0800] "GET /robots.txt HTTP/1.0" 200 101 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.86.122 - - [13/Dec/2002:21:48:04 -0800] "GET /robots.txt HTTP/1.0" 200 101 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.86.54 - - [13/Dec/2002:21:53:51 -0800] "GET /robots.txt HTTP/1.0" 200 101 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.82.58 - - [13/Dec/2002:23:53:58 -0800] "GET /robots.txt HTTP/1.0" 200 101 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"

Other pages stripped...

Perhaps each cluster needs to download the robots.txt file. But I think they could share...

[pendanticist]

I see. Roughly how many were in between each request?
Pendanticist.

[mack]

OK I read you wrong I thought it was requesting pages that you didnt have...lol I need to wake up

It has happened to me once last month. I think it was fresh bot. seamed to be making a lot of requests for the same page.

are you using sessions?

[pendanticist]

are you using sessions?

Don't know what they are.

Anyway, it's nappy time. I'll check back on the morrow.

Pendanticist.

[Lisa]

Maybe two or three pages between requests. It is like each IP that Googlebot uses needs its own robots.txt file.

[pendanticist]

Understood Lisa,

Sure is an odd thing, isn't it?

Pendanticist.

[pendanticist]

Update

Google musta had a loooooooong dink a wa-wa because all seems to be better today.

Pendanticist.

[cfx211]

I have seen this before in the past, but mostly when the googlebot was in our dynamic content. I think that it gets nervous when it starts to see a lot of query string variables and checks the robots.txt to make sure that it is really ok to be in this section of the site.