Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- Digsby IM Enables Web Crawlers Control of Your PC & Bandwidth


incrediBILL - 10:35 pm on Sep 8, 2009 (gmt 0)


Now to throw another wrinkle into the fray...

Does 80legs plan to offer a way for webmasters to validate that it's actually 80legs making the crawl and not someone faking or spoofing their user agent?

Majestic12, another distributed crawler, recently announced a mechanism to allow webmasters to validate MJ12bot [webmasterworld.com], so what say you 80legs?

And finally the most important thing... Digsby is OPT IN. It sounds like they were a bit sneaky in terms of hiding it in the fine print, but now that they have stopped doing that I don't see the problem.

Yahoo's toolbar is opt-in too and I've accidentally installed it a few times skipping through an install in a hurry.

Besides, most users aren't savvy enough to know what's going on half the time, the computer is an appliance and all that techno babble just confuses them so they tend to click OK just to make the box go away.


Thread source:: http://www.webmasterworld.com/search_engine_spiders/3986022.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com