Googlebot is going nuts and it's killing my site!

300 simultaneous connections?!

[Rearden]

For the past week I've had hours of downtime because of Googlebot completely thrashing my site. It is in a directory format with thousands of links, and that infernal bot has suddenly started opening 250-300 simultaneous connections and making over 5 requests per second.

This has only begun in the past week, even though the side has been up for 6 months and has been being spidered all along. A colleague who runs a similar type of site has also had downtime this week for the same reason. Has Google started using more aggressive spidering techniques? Is there anything I can do to limit Google's activity while still letting it index us? It'd be nice if robots.txt had a provision for a "slow the heck down!" tag.

[WebGuerrilla]

If you find that we are placing too high a load on your site, please let us know by sending us e-mail at [e-mail].

I would start there.

[Jesse_Smith]

Do you have session IDs in the URL? If you do, get rid of them! They only either get you Googlebombed, or give you the Google death penalty.

[futureX]

maybe your PR went up and google decided to give you a [bit] more attention :)