Page is a not externally linkable
incrediBILL - 10:35 pm on Sep 8, 2009 (gmt 0)
Does 80legs plan to offer a way for webmasters to validate that it's actually 80legs making the crawl and not someone faking or spoofing their user agent? Majestic12, another distributed crawler, recently announced a mechanism to allow webmasters to validate MJ12bot [webmasterworld.com], so what say you 80legs? Yahoo's toolbar is opt-in too and I've accidentally installed it a few times skipping through an install in a hurry. Besides, most users aren't savvy enough to know what's going on half the time, the computer is an appliance and all that techno babble just confuses them so they tend to click OK just to make the box go away.
Now to throw another wrinkle into the fray... And finally the most important thing... Digsby is OPT IN. It sounds like they were a bit sneaky in terms of hiding it in the fine print, but now that they have stopped doing that I don't see the problem.