Page is a not externally linkable
Play_Bach - 1:41 am on Nov 24, 2005 (gmt 0)
In reading this thread I can't help but wonder how does eBay or Amazon avoid the bot problem, assuming it's as much a threat to them as it is to WebmasterWorld? Also, as far as bandwidth use goes, Yahoo! has something in place that prevents downloading/archiving too much from any IP (once exceeded, Yahoo! returns a "Yahoo Error Unable to process request at this time -- error 999" page which can last several hours to a day). Maybe something like that could be installed. Perhaps consulting with the IT of some of the major portals is something you've already done, if not, maybe there's some good ideas to be found there? Just a thought... Best wishes.
Hi Brett,