Forum Moderators: open

Message Too Old, No Replies

Odd Googlebot behavior

hundreds of requests for robots.txt

         

Rugles

1:18 pm on Sep 25, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I have never seen this before. Usually my favorite little bot comes in from multiple IP addresses, today only one IP. Plus it has requested the robots.txt file a couple of hundred times. This has all occured in the last 5 hours. It has also grabbed a couple thousand pages.

Has anyone ever seen this before?
Is googlebot stuck?

Rugles

4:20 pm on Sep 25, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



As many as 3 pages a second and it temporarily crashed the server.

Google Guy..... whats up?

SEOPTI

5:54 pm on Sep 25, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



It crashed you machine? What server do you use a 50MHz PI machine <lol>

Rugles

6:45 pm on Sep 25, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



No it is a little better that that.

By the time the bot left it had racked up 7500 hits on the server in just a few hours. Why the server crashed we do not know yet.

I get daily visits from googlebot but it has never been so aggressive.

Oh ya... there is multilple sites on the server and it had quite a traffic spike this morn.

bobmark

7:42 pm on Sep 25, 2002 (gmt 0)

10+ Year Member



Definitely weird!
you wonder if it was the other way around: if some server malfuncion caused Googlebot to loop, not Googlebot caused the server to crash.