Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- Digsby IM Enables Web Crawlers Control of Your PC & Bandwidth


shiondev - 5:15 pm on Sep 8, 2009 (gmt 0)


incrediBill,

I work on 80legs. I thought I should point out a few things that affect your analysis.

1. The number of computers in the distributed grid actually fluctuates. 50,000 is the average number we have seen connected at any given time, but it can be as high as 200,000 during certain points in the day.

2. We have built bandwidth-monitoring technology into our crawler. 80legs will never use more bandwidth than what a given computer's bandwidth cap is. We keep up-to-date records on current ISP bandwidth plans and caps and only use computers that are using an ISP for who we know the plan. We will never use a computer in a way that risks going over the cap.

3. Digsby has changed their install process since the Lifehacker article was posted. It is much easier to disable Plura now. They are also working on an entirely new installer that will show Plura during the install process, making it immediately apparent. Granted, it would be good if this is done asap, but from what I understand they have to work through several business and technical issues to make this happen.

4. Our user-agent 008, obeys robots.txt, so webmasters can control access to their sites.


Thread source:: http://www.webmasterworld.com/search_engine_spiders/3986022.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com