Googlebot download speed.

GB is hitting too much causing site to go down

[sadierae]

I have a client whose site is getting hit by Googlebot 40-50 times a min. and is causing their site to go down. GB seems to be going into their search box performing a search querry and hitting their databases- the problem is that its clogging up the port- crashing the site.

Its not a problem that Google is indexing a lot of their dynamic product pages but how can we get Googlebot not to hit so many times?Is there and easy fix to get GB to "slow" down with crawling(not stop)?

[EliteWeb]

Ive never had googlebot 'search' or submit or try to even access any form function. Check to see that the IP it is showing up from is googles IP and that there isnt a page on the server which would querie or if google found it clicked on the link ie: search.cgi?item=search1+search2 type thing. And if the site cant handle 40-50 requests a minute its time to look into a different way to program the site so it doesnt Denial of Service attack the computer to bring it down to its knees.

[teeceo]

"GB to "slow" down with crawling(not stop)?"

first of all, are you "INSANE"? Stopping googlebot from crwaling is a capital offence, punishable by pr0. If you don't want the freshbot, i'll take him, as you see, I have a dedictated server that doesn't mind googlebot at all. Send the googlebot to www.i-dont-know-what-to-do-with-the-goldmine-name-googlebot.whatever. Last thought(how dare you).

teeceo.

[edited by: teeceo at 9:08 pm (utc) on Jan. 8, 2003]

[Yidaki]

If your clients search url isn't disallowed for robots (check robots.txt) it's time to do it!

However, like Elite said, 40-50 hits p. min really isn't much ... take a look at how the site's programmed! (Allthough it's too much for googlebot. Normally she's doesn't hit that often.)

If you don't find the reason you should look at the google's faq Help! Googlebot is crawling my site too fast. What can I do? [google.com]!

Please send an email to [e-mail] with the name of your site and a detailed description of the problem. Please also include a portion of the weblog that shows Google accesses, so we can track down the problem more quickly on our end.

[sadierae]

They will be switchig programs shortly however I think it is doing something very strange. I was going to send an email to Google just wanted to check here first!

Teeco, to your reply "first of all, are you "INSANE"? Stopping googlebot from crwaling is a capital offence, punishable by pr0."- EXACTLY- I dont want to stop GB just slow it down a little. And too: "Last thought(how dare you)"- Can't have a site the is going down all the time- it makes no money that way!;)

[atadams]

I don't know how Googlebot could fill out and submit a form, but it was following some link on my site that were causing a backend app to overload. I updated my robots.txt and put the links to the backend app in a Javascript.

[jomaxx]

I agree with the preceding advice - fix your code and make it harder for spiders to invoke scripts. Remember that there are plenty of other personal web bots that behave much worse than Googlebot. Ever been crawled at the rate of 40 or 50 pages a SECOND? Ouch.

[redzone]

Googlebot is actually very "server friendly" these days, compared to their older spider systems back in 1999-2000.

40-50 requests per minute is minute, compared to their 100-300 requests per second, they used to fire off! Now that was enough volume to strain, even a dual processor box, back in those days.... LOL

[espeed]

The max requests per second (RPS) a server can handle obviously depends on the server, but for example, AOLserver running on a PIII 500 MHz with 512 MB can do ~30 RPS for DB pages and more than 1000 RPS for static pages -- 50 requests per min is not much and shouldn't cause server problems.

One of my sites has >45K pages, and googlebot visits it 24x7 at about 30 pages per min -- how big is the site that's getting 50 pages per min?