Page is a not externally linkable
- Local
-- Foo
---- Attack of the Robots, Spiders, Crawlers.etc


notsleepy - 10:03 pm on Nov 27, 2005 (gmt 0)


> These are highly knowledgeable, highly motivated, and proactive visitors who are not new to site ripping.

I see. I didn't realize that the bulk of the problem were from skilled rippers. Honestly its easy to get around any deterrent including Google's, Yahoo's and Jeeve's. (I still have trouble believing that these highly knowledgeable, ruthless rippers are obeying robots.txt). In this case the only solution is load balancing. Hire an architect, gutt BestBBS, and rebuild with a DB solution. Yes its hard, and yes its a lot of work, but bandaids won't cut it anymore.


Thread source:: http://www.webmasterworld.com/foo/9618.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com