jmccormac - 5:27 pm on May 8, 2010 (gmt 0)

@ByronM The web has changed a lot since this discussion: [webmasterworld.com...] and many of the AWS maggots seem to be oblivious to the existence of robots.txt. Now when you run a small website, a few pages here or there is very little. But when you run a large website with thousands or millions of webpages and the operators of some of these maggots decide to download the entire site, it is a big problem.

Regards...jmcc