Heavy GoogleBot Spidering!

DOS Attack from the Googlebot!

[Jesse_Smith]

Has any one ever had there server crash from the Googlebot doing a nice DOS attack on the server?! The Googlebot has been chomping away at my Archive version of my vBulletin boards and took the server load up to 24.56, 10.49, 5.14. For a few minutes it it slowed the server down and looked like it would crash! Now it's back down to 1-3!

[ciml]

There's always /robots.txt :-)

I used to let the bots have flat HTML files, while people with session data or customisations would get the live content at slightly different URLs.

IMO hardware is cheap enough now that it normally shouldn't matter. If the bots are eating significantly more resources than the human visitors they're not sending enough people to warrant their visits.

[Jesse_Smith]

Banning it is the last thing I would do, since it's not always doing it, only slowed the server down for about five minutes, and Google brings in A LOT of visitors! With the vBulletin archive hack added last month, I expected to get attacked.