spiders moving too quickly

41 queries per second

[JonR28]

We have a site that uses a lot of queries which is no problem with usual visitor load, but when the spiders come through (for some reason they like to do it all at once) they pretty much wipe out the CPU doing 41 queries per second! I know this is a good thing for getting our pages indexed, but is there anyway to tell spiders to slow down a bit? We can't afford to upgrade the server, its already a 2.2Ghz Xenon with a gig of ram.

Soon we will be implementing more efficient code but until then does anyone have any suggestions? Thanks so much!

[wilderness]

41 queries per second

Are these major bots grabbing at this speed or perhaps undesireables?

My sites don't have Google, MSN, Yahoo or Jeeves grabbing pages at this rate and in the wee hours hours of the A.M.

My "personal" preference is for bots to grab all and be done, rather than the very slow crawing methods of which I see cluttering my logs daily. (My pages however are primarily text and in most instances (even though the text may be lengthy)page sizes are less than 50kb.

Don

[JonR28]

They are major bots and they seem to hit right in the middle of the day.

[surfin2u]

I have never seen a major bot grab pages faster than 2 or 3 a minute, not the 640 a minute that you mentioned. Major bots scan my site every day.

I have also been hit by "site hogs" that accessed huge numbers of pages. I've written some code to prevent anyone from grabbing more than one page per second from my site.

Later, when I check the site hog's IP address, they usually turn out to be a cable company. Maybe a salesman showing off how fast they are or someone bored with nothing better to do?

[trillianjedi]

Check you're not serving the bots session ID's.....

TJ