Page is a not externally linkable
- Local
-- Foo
---- Attack of the Robots, Spiders, Crawlers.etc


Kirby - 4:41 am on Dec 1, 2005 (gmt 0)


I agree with Oil in alot ways, but - like alot of other guys - he isn't getting the scale and scope of the situation. It isn't just server management, page views, bandwidth, or servers. There are also issues of scrapping, copyright, and liability. The site is here for the human members - not bots.

More of us get it than you think. So lets assume you cannot win the war of the bots. What is the acceptable loss range? Scraping and copyright, no search feature, what? Does the host die to kill the parasite?


Thread source:: http://www.webmasterworld.com/foo/9618.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com