AlexK - 1:46 pm on Dec 27, 2011 (gmt 0)
Another day, and a 10th MSNBot IP committing abuse upon my site:
2011-12-26 05:47:33 :: 22.214.171.124 [forums.modem-help.co.uk] :: max 12 pages / second
Do not pay too much attention to some IPs getting stopped with a 403, and some with a 503. In brief, it works like this:
There are two tests:
1) Fast scraper (>= 3 page / sec)
2) Slow scraper (forums only; > 50 pages in 1 hour)
Only fast scrapers are reported.
Once spotted, a fast scraper is given a 403 block.
If a fast scrape continues for long enough, it will be caught first by the slow-scrape routine & given a 503.