-- Search Engine Spider and User Agent Identification
---- Best method of blocking?
jmccormac - 12:19 pm on Aug 30, 2013 (gmt 0)
Is this 40k in ranges or 40k in file size?
If 40k in ranges than you have the IP tables complied incorrectly (at least in regards to IP ranges).
40K in IP ranges. I have the website:ip address mapping of websites in com/net/org/biz/info/mobi/asia/us and approximately another fifteen million sites in various ccTLDs. It is part of a survey that I run. I suppose I could optimise the ranges. What I have noticed is a lot of Chinese and Indian subnets using US and CA IP ranges.