scrapyproject

[keyplyr]

UA: scrapyproject (+http://www.profound.net)
Protocol: HTTP/1.1
Robots.txt: No
Host: AWS
50.112.0.0 - 50.112.255.255
50.112.0.0/16

B2B Marketing

[keyplyr]

Seems to be entirely different from...

UA: Scrapy/1.0.5 (+http://scrapy.org)
Protocol: HTTP/1.1
Robots.txt: No
Host: hugeserver.com
192.173.144.0 - 192.173.159.255
192.173.144.0/20

[blend27]

profound.net is also in:

Mozilla/5.0 (compatible; DomainAppender /1.0; +http://www.profound.net/domainappender)

Also comes from all over Amazon AWS

[keyplyr]

How profound

[lucy24]

profound.net is also in

Ah, DomainAppender, rapidly approaching a place on my "when hell freezes over" list because each visit consists of three (3) robots.txt requests-- where they are denied-- followed regardless by one (403) request for front page.

Anyone know what "domain appender" actually means-- or even what it is intended to make you think it means?

[blend27]

Lucy,

I think the link in that UA would explain what they are trying accomplish. It also mentions that they obey robots.txt, well that is complete bullocks as I have seen the same exact behavior as you mentioned on over 2 dozen sites so far.

I ended up simply issuing AbortRequest in web.config <rewrite rules> based on "profound" string because to me this is simply a referrer spam, be it a legitimate biz or not.... so I don't even have to see it the log entries.