Welcome to WebmasterWorld Guest from

Message Too Old, No Replies

Beware of Asking Google to Slow Their Mozilla Compatible Robot Down?



1:56 pm on Jul 6, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

On 28 June G told me "We've reduced the load on your servers".

The M_Bot [webmasterworld.com] (identified by

Mozilla/5.0 (compatible; Googlebot/2.1;
in the referer string) had been hitting my site at upto 3 times/sec (avg: 836/day), triggering unruly-bot-prevention routines [webmasterworld.com] in the PHP-scripts. I had used the G on-site form [google.com] a few days before, and asked them (nicely) to stop it. So far so good.

By July 1 it was clear that--rather than turning the knob down a notch or two--they had switched it off altogether, at least as far as the M_Bot was concerned - there were no hits from this bot at all. The G_Bot (identified just by

Googlebot/2.1 (+http://www.google.com/bot.html)
in the referer string) had also slowed:
    June avg: 60/day
    Jul 1: 12
    Jul 2: 18
    Jul 3: 5
    Jul 4: 17
    Jul 5: 29

So, my site has now dropped from 28,000 G-hits in June to a likely 500 in July. So, beware...


11:15 pm on Aug 21, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

Now there is a third thread on the same subject to add to the previous two:

That is now 4 threads (including this one) with numerous people reporting excessive bandwidth.

It is worth reporting that the number of pages resulting from G-SERPs on my site is actually up:

  • July pages/day = 2,019
  • Aug pages/day = 2,034
...and here are the pages taken so far in Aug by the different bots:
  • Inktomi Slurp : 16,495
  • Google AdSense : 10,473
  • MSNBot : 8,663
  • Googlebot HTTP/1.0 : 3,534
  • Googlebot-Image : 198
  • Googlebot HTTP/1.1 Mozilla/5.0 : 72
So far, slowing the M_Bot has cost my site nothing, and kept the bandwidth clear for human visitors. Time will tell if this continues.


12:57 am on Sep 14, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

Some interesting extra info on the M_Bot, thanks to zCat [webmasterworld.com], and confirmed on my site:- this bot does NOT implement the If-Modified-Since header (nor does the G-MediaPartners-Bot), in spite of GoogleGuy's advice to Webmasters [webmasterworld.com] back in October 2002. That nicely adds depth to why this wretched bot is such a drain on our server's resources. Or at least, on your server's resources, since it hardly ever hits my site now.

To rub salt in my wounds, all M_Bot hits on my site in Sept are 301s and 404s for pages that have never existed on my site, ever. Here are 3 typical examples:

GET /topsites.php?ID=259 HTTP/1.1" 404 610
"GET /out.php?ID=1 HTTP/1.1" 404 610
"GET /index.shtml HTTP/1.1" 404 610

Here are the number of pages/day from G on my site:

  • Jun pages/day = 2,270
  • July pages/day = 2,019
  • Aug pages/day = 2,002
  • Sep pages/day = 1,599
(which begins to look like slow death)

...and the number of pages to G:

  • Jun pages/day = Adsense: 514; G_Bot: 60; M_Bot: 836; Image: 0
  • July pages/day = Adsense: 515; G_Bot: 28; M_Bot: 2; Image: 3
  • Aug pages/day = Adsense: 537; G_Bot: 29; M_Bot: 4; Image: 9
  • Sep pages/day = Adsense: 873; G_Bot: 40; M_Bot: 0; Image: 0

Seeing the slow death of hits from G has shifted something in my attitude to Google. I can no longer be counted as one of their supporters. And, whilst that statement is unlikely to cause G's knees to tremble, I think that maybe it should. We all need all the help that we can get from other people of goodwill, and I think that it is a shame that such intelligent people should, in the end, prove to be so ignorant.


2:34 pm on Sep 23, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

which begins to look like slow death

...or not so slow.

Pages supplied from Google halved yesterday. No-one else reported a sudden change, so it must be personal to my site.

Here are stats for September; 1st column is upto and including 21st, 2nd column is 22nd:

Search Avg 22nd 
------ ---- ----
Google 1650 824
Yahoo. .234 231
MSN... ..90 .91

<edit>In fact, this thread [webmasterworld.com] is full of cries-of-pain from folks with sites that have suddenly tanked on 20/21st Sep.</edit>
This 33 message thread spans 2 pages: 33

Featured Threads

Hot Threads This Week

Hot Threads This Month