dstiles - 10:19 pm on Jan 5, 2011 (gmt 0)
After adding mj12 to robots.txt the spate of bots died down and finally ceased, at least on the web pages (I assume robots.txt is still hit occasionally).
I also now notice an absence of "fake" mj12 bots which prompts me to wonder:
a) were they really fake or simply out of date and out of control; OR
b) did the drivers of the fakes simple lose interest; OR
c) were the bots themselves gradually updated by the fakers (who in that case were running genuine mj12's).
I doubt very much the mj12's were (eg) renamed nutch bots or we would still be seeing them
Credit to Majestic for obeying robots.txt. Nevertheless, I still will not permit distributed bots on my server. There is far too little control over them OR too much hassle setting up for them (eg mj12's validation).