motorhaven - 10:25 pm on Nov 28, 2005 (gmt 0)

WW Alexa ratings have plunged since this started. We'll have to see how long past Thanksgiving this trend continues. Given the number of webmasters with Alexa's plugin installed the chart, while not accurate comparing one site to another, does give strong indications of the trends of a single site. Plus robots/spiders/crawlers don't use it so this shows only true visitor trends.

Looks like about a sudden 60 percent drop in WW user traffic.

I run a very high volume site with forums. Over 1 terabyte of bandwidth per month. I can say from my own experience there are better ways to deal with this while still allowing legit search engine crawlers in.

Our intelligent automated approach works well:

1. Dynamic robots.txt
2. HTTP honey-pot without instant auto-blocking.
3. Honey-pot from projecthoneypot to catch the sneakiest of crawlers.
4. Automated log file scanning.
5. Firewalling of problematic IP ranges (not just single IPs). If I have to lose a handful of visitors in order to gain many more legit visitors from SERPs, so be it!
6. Lastly, hardware and bandwidth is cheap... throw more at it if you can. Its less costly than 8 hours per week of your time.

We're going to be adding a geotargetting database soon, once we get another server up.