Big_Gig - 3:09 pm on Jun 29, 2005 (gmt 0)

Hey guys,

I recently had this problem too. They spidered thousands of my pages in one day - totally took down the dedicated server!

I found this piece of code on their site which has worked well for me. It tells the bot to wait at least 30 seconds inbetween page requests:

User-agent: BecomeBot
Crawl-Delay: 30
Disallow: /cgi-bin