Page is a not externally linkable
AlexK - 2:09 am on Feb 17, 2006 (gmt 0)
In January it took 60 pages, 69 in December, and very similar numbers in previous months. Then yesterday it took 515 pages up until 21:56:36, when the site Slow-Scaper block [webmasterworld.com] kicked in (set at 1,000 pages), and the bot started getting 503's (it was sharing the IP with the A-Bot--also going mad--and they both got blocked). I recently implemented [webmasterworld.com] the Retry-After header for 503 Responses, but both bots have ignored it. Here is an edited selection of timings, all for the G Mozilla bot only (the last is a 503, all others are 200, 304 or 301):
At 06:26:28 on 16 Feb the M-Bot began to go mad on my site. 66.249.66.147 - - [16/Feb/2006:00:20:47 +0000] "GET /mfcs.php?mid=25 HTTP/1.1" 200 12038 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" In:107461 Out:12038:11pct.
66.249.66.147 - - [16/Feb/2006:05:23:00 +0000]
66.249.66.147 - - [16/Feb/2006:06:26:28 +0000]
66.249.66.147 - - [16/Feb/2006:06:26:30 +0000]
66.249.66.147 - - [16/Feb/2006:06:26:32 +0000]
66.249.66.147 - - [16/Feb/2006:06:26:33 +0000]
66.249.66.147 - - [16/Feb/2006:06:26:34 +0000]
...
66.249.66.147 - - [16/Feb/2006:21:56:32 +0000]
66.249.66.147 - - [16/Feb/2006:21:56:34 +0000]
66.249.66.147 - - [16/Feb/2006:21:56:35 +0000]
66.249.66.147 - - [16/Feb/2006:21:56:36 +0000]
66.249.66.147 - - [16/Feb/2006:21:56:38 +0000] "GET /mfcs.php?mid=111&nid=7475 HTTP/1.1" 503 162 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" In:191 Out:144:75pct.
.
.
# egrep -c "66\.249\.66\.147 - - \[16/Feb(.*) [23]0[014] (.*)Mozilla/5.0" access_log
515