joined:Jan 20, 2005
So the past few days have seen a massive increase in traffic which looks legit based on IP address but it seems that in fact we have been hit over 80,000 times in the past 48 hours by 80legs!
Who would have thought that <10,000 pages of content would interest the bot that much.
Regardless, blocked them with robots.txt 24 hours ago to noavail, forcing me to block any and all user agents with preg_match '/80legs/i' true :-(
On another thread here on WW back in 2010, many expressed concerns that they might not honour robots.txt in the future; seems like they were correct.