| This 33 message thread spans 2 pages: < < 33 ( 1  ) || |
|Beware of Asking Google to Slow Their Mozilla Compatible Robot Down?|
| 1:56 pm on Jul 6, 2005 (gmt 0)|
On 28 June G told me "We've reduced the load on your servers".
The M_Bot [webmasterworld.com] (identified by Mozilla/5.0 (compatible; Googlebot/2.1; in the referer string) had been hitting my site at upto 3 times/sec (avg: 836/day), triggering unruly-bot-prevention routines [webmasterworld.com] in the PHP-scripts. I had used the G on-site form [google.com] a few days before, and asked them (nicely) to stop it. So far so good.
By July 1 it was clear that--rather than turning the knob down a notch or two--they had switched it off altogether, at least as far as the M_Bot was concerned - there were no hits from this bot at all. The G_Bot (identified just by Googlebot/2.1 (+http://www.google.com/bot.html) in the referer string) had also slowed:
June avg: 60/day
Jul 1: 12
Jul 2: 18
Jul 3: 5
Jul 4: 17
Jul 5: 29
So, my site has now dropped from 28,000 G-hits in June to a likely 500 in July. So, beware...
| 11:15 pm on Aug 21, 2005 (gmt 0)|
Now there is a third thread on the same subject to add to the previous two:
That is now 4 threads (including this one) with numerous people reporting excessive bandwidth.
It is worth reporting that the number of pages resulting from G-SERPs on my site is actually up:
...and here are the pages taken so far in Aug by the different bots:
- July pages/day = 2,019
- Aug pages/day = 2,034
So far, slowing the M_Bot has cost my site nothing, and kept the bandwidth clear for human visitors. Time will tell if this continues.
- Inktomi Slurp : 16,495
- Google AdSense : 10,473
- MSNBot : 8,663
- Googlebot HTTP/1.0 : 3,534
- Googlebot-Image : 198
- Googlebot HTTP/1.1 Mozilla/5.0 : 72
| 12:57 am on Sep 14, 2005 (gmt 0)|
Some interesting extra info on the M_Bot, thanks to zCat [webmasterworld.com], and confirmed on my site:- this bot does NOT implement the If-Modified-Since header (nor does the G-MediaPartners-Bot), in spite of GoogleGuy's advice to Webmasters [webmasterworld.com] back in October 2002. That nicely adds depth to why this wretched bot is such a drain on our server's resources. Or at least, on your server's resources, since it hardly ever hits my site now.
To rub salt in my wounds, all M_Bot hits on my site in Sept are 301s and 404s for pages that have never existed on my site, ever. Here are 3 typical examples:
|GET /topsites.php?ID=259 HTTP/1.1" 404 610 |
"GET /out.php?ID=1 HTTP/1.1" 404 610
"GET /index.shtml HTTP/1.1" 404 610
Here are the number of pages/day from G on my site:
(which begins to look like slow death)
- Jun pages/day = 2,270
- July pages/day = 2,019
- Aug pages/day = 2,002
- Sep pages/day = 1,599
...and the number of pages to G:
- Jun pages/day = Adsense: 514; G_Bot: 60; M_Bot: 836; Image: 0
- July pages/day = Adsense: 515; G_Bot: 28; M_Bot: 2; Image: 3
- Aug pages/day = Adsense: 537; G_Bot: 29; M_Bot: 4; Image: 9
- Sep pages/day = Adsense: 873; G_Bot: 40; M_Bot: 0; Image: 0
Seeing the slow death of hits from G has shifted something in my attitude to Google. I can no longer be counted as one of their supporters. And, whilst that statement is unlikely to cause G's knees to tremble, I think that maybe it should. We all need all the help that we can get from other people of goodwill, and I think that it is a shame that such intelligent people should, in the end, prove to be so ignorant.
| 2:34 pm on Sep 23, 2005 (gmt 0)|
|which begins to look like slow death |
...or not so slow.
Pages supplied from Google halved yesterday. No-one else reported a sudden change, so it must be personal to my site.
Here are stats for September; 1st column is upto and including 21st, 2nd column is 22nd:
Search Avg 22nd
------ ---- ----
Google 1650 824
Yahoo. .234 231
MSN... ..90 .91
<edit>In fact, this thread [webmasterworld.com] is full of cries-of-pain from folks with sites that have suddenly tanked on 20/21st Sep.</edit>
| This 33 message thread spans 2 pages: < < 33 ( 1  ) |