Forum Moderators: open

Message Too Old, No Replies

zermelo

         

zerillos

10:01 am on Mar 4, 2008 (gmt 0)

10+ Year Member Top Contributors Of The Month



I got this today from 67.202.4.150

Agent: zermelo Mozilla/5.0 compatible; heritrix/1.12.1 (+http://www.powerset.com) [email:crawl@powerset.com,email:paul@page-store.com]

At that URL they say : "Natural Language Search". I don't know why but I kind of doubt that.

Any other opinions?

To block or not or block?

wilderness

6:41 pm on Mar 4, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



There are hordes of references to the IP range and backbone provider.

The following portion of their UA, falls into at least three of my OWN denied terms.

[email:crawl@powerset.com,email:paul@page-store.com]

incrediBILL

8:34 pm on Mar 19, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



That's coming from the Amazon Web Services where they rent out some computer time to various developers for all sorts of stuff.

The key word in the UA is "heritrix" which is the open source that came from ia_archive, somewhat like nutch in the fact that there are hundreds of them out there crawling.