bitbucket - 2:38 pm on Jun 15, 2012 (gmt 0)
These folks just hit my org yesterday, about 6k hits over the 2 hours before I completely blocked them. Massively overloaded our DB and killed the sites. I did use the robots.txt mod as suggested on their site, but I also collected approximately 850 IP's (all overseas) from my web logs and blocked them via iptables rules in my firewall.
I did have a filter setup in my apache server to reject queries by specified agents, however it had no effect. It wasn't a crawler as much as a botnet based DDoS. IMHO - if you see these guys in your logs, block them.