Beware of Asking Google to Slow Their Mozilla Compatible Robot Down?

[AlexK]

On 28 June G told me "We've reduced the load on your servers".

The M_Bot [webmasterworld.com] (identified by

Mozilla/5.0 (compatible; Googlebot/2.1;

in the referer string) had been hitting my site at upto 3 times/sec (avg: 836/day), triggering unruly-bot-prevention routines [webmasterworld.com] in the PHP-scripts. I had used the G on-site form [google.com] a few days before, and asked them (nicely) to stop it. So far so good.

By July 1 it was clear that--rather than turning the knob down a notch or two--they had switched it off altogether, at least as far as the M_Bot was concerned - there were no hits from this bot at all. The G_Bot (identified just by

Googlebot/2.1 (+http://www.google.com/bot.html)

in the referer string) had also slowed:

June avg: 60/day
Jul 1: 12
Jul 2: 18
Jul 3: 5
Jul 4: 17
Jul 5: 29

So, my site has now dropped from 28,000 G-hits in June to a likely 500 in July. So, beware...

[AlexK]

Now there is a third thread on the same subject to add to the previous two:

• 28957 hits from Google's Robot in 2 weeks? [webmasterworld.com] (18 Aug) July=64 hits; MSN=357 in Aug.
  Total pages=<100. No increase in traffic.
  (Aug 20): 50,249 hits.

That is now 4 threads (including this one) with numerous people reporting excessive bandwidth.

It is worth reporting that the number of pages resulting from G-SERPs on my site is actually up:

• July pages/day = 2,019
  
• Aug pages/day = 2,034

...and here are the pages taken so far in Aug by the different bots:

• Inktomi Slurp : 16,495
  
• Google AdSense : 10,473
  
• MSNBot : 8,663
  
• Googlebot HTTP/1.0 : 3,534
  
• Googlebot-Image : 198
  
• Googlebot HTTP/1.1 Mozilla/5.0 : 72

So far, slowing the M_Bot has cost my site nothing, and kept the bandwidth clear for human visitors. Time will tell if this continues.

[AlexK]

Some interesting extra info on the M_Bot, thanks to zCat [webmasterworld.com], and confirmed on my site:- this bot does NOT implement the If-Modified-Since header (nor does the G-MediaPartners-Bot), in spite of GoogleGuy's advice to Webmasters [webmasterworld.com] back in October 2002. That nicely adds depth to why this wretched bot is such a drain on our server's resources. Or at least, on your server's resources, since it hardly ever hits my site now.

To rub salt in my wounds, all M_Bot hits on my site in Sept are 301s and 404s for pages that have never existed on my site, ever. Here are 3 typical examples:

GET /topsites.php?ID=259 HTTP/1.1" 404 610
"GET /out.php?ID=1 HTTP/1.1" 404 610
"GET /index.shtml HTTP/1.1" 404 610

Here are the number of pages/day from G on my site:

• Jun pages/day = 2,270
  
• July pages/day = 2,019
  
• Aug pages/day = 2,002
  
• Sep pages/day = 1,599

(which begins to look like slow death)

...and the number of pages to G:

• Jun pages/day = Adsense: 514; G_Bot: 60; M_Bot: 836; Image: 0
  
• July pages/day = Adsense: 515; G_Bot: 28; M_Bot: 2; Image: 3
  
• Aug pages/day = Adsense: 537; G_Bot: 29; M_Bot: 4; Image: 9
  
• Sep pages/day = Adsense: 873; G_Bot: 40; M_Bot: 0; Image: 0

Seeing the slow death of hits from G has shifted something in my attitude to Google. I can no longer be counted as one of their supporters. And, whilst that statement is unlikely to cause G's knees to tremble, I think that maybe it should. We all need all the help that we can get from other people of goodwill, and I think that it is a shame that such intelligent people should, in the end, prove to be so ignorant.

[AlexK]

which begins to look like slow death

...or not so slow.

Pages supplied from Google halved yesterday. No-one else reported a sudden change, so it must be personal to my site.

Here are stats for September; 1st column is upto and including 21st, 2nd column is 22nd:

Search Avg 22nd 
------ ---- ---- 
Google 1650 824 
Yahoo. .234 231 
MSN... ..90 .91

<edit>In fact, this thread [webmasterworld.com] is full of cries-of-pain from folks with sites that have suddenly tanked on 20/21st Sep.</edit>