Forum Moderators: DixonJones
SBIder/0.8-dev (SBIder; [sitesell.com...] [support.sitesell.com...]
The spider page says it's Nutch based and respects robots.txt; as to what it does, they say
SiteSell is gathering a statistical representation of topics presented on the Web as a whole. Each Web page visited is categorized under the topics that it represents, allowing our customers to know the percentage of Web pages that are about any particular topic.
Presumably only SiteSell customers benefit from this; is there any reason not to block them?