PrivacyAwareBot

[w3bmastine]

UA: PrivacyAwareBot/1.1; +http://www.privacyaware.org
Protocol: HTTP/1.1
Robots.txt: No
Host: Yomura Corporation?
107.189.60.80

Does not fetch robots.txt although robots page states differently. Only two requests...

107.189.60.80 - - [09/Feb/2016:13:27:17 +0100] "HEAD / HTTP/1.1" 403 0 "-" "Mozilla/5.0 (compatible; PrivacyAwareBot/1.1; +http://www.privacyaware.org)" "example.com"
107.189.60.80 - - [09/Feb/2016:13:27:17 +0100] "GET / HTTP/1.1" 200 793 "-" "Mozilla/5.0 (X11;) Firefox/38.0" "example.com"

[keyplyr]

@w3bmastine - I had a different UA visit my client's EU based site:
Mozilla/5.0 (compatible; PrivacyAwareBot/1.1; +http://www.privacyaware.org)

If you wish to block our bot you can do so using the standard robots.txt standard. However, if you do we will report you as an unknown site, that may be breaching data best practices.

Yawn...

[keyplyr]

First I've actually seen it in logs. Head check, then pretended it was Firefox, then grabbed all the scripting I have in the head section, presumably to discover set-cookies or cloaking or...

107.189.60.80 - - [27/Feb/2016:11:18:05 -0800] "HEAD / HTTP/1.1" 200 353 "-" "Mozilla/5.0 (compatible; PrivacyAwareBot/1.1; +http://www.privacyaware.org)"
107.189.60.80 - - [27/Feb/2016:11:18:05 -0800] "GET / HTTP/1.1" 200 8280 "-" "Mozilla/5.0 (X11;) Firefox/38.0"

[wilderness]

Yomura Corporation (YC-5) (also goes by other names and/or domains (delimiter.us))
YOMURA-CORE-001 107.189.32.0 - 107.189.63.255 107.189.32.0/19
YOMURA-WEST-001 199.204.184.0 - 199.204.187.255 199.204.184.0/22
YOMURA-EAST-001 199.233.244.0 - 199.233.247.255 199.233.244.0/22
YOMURA-CORE-001 2605:980:: - 2605:980:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF

[lucy24]

PrivacyAwareBot

Can the IronyBot be far behind?

:: detour to raw logs for case-insensitive search for "Privacy" in UA ::

Oh yes indeed.

107.189.60.76 - - [14/Nov/2015:06:43:14 -0800] "HEAD / HTTP/1.1" 403 164 "-" "Mozilla/5.0 (compatible; PrivacyAwareBot/1.1; +http://www.privacyaware.org)" 
107.189.60.79 - - [08/Feb/2016:13:09:10 -0800] "HEAD / HTTP/1.1" 403 193 "-" "Mozilla/5.0 (compatible; PrivacyAwareBot/1.1; +http://www.privacyaware.org)" 
107.189.60.65 - - [11/Feb/2016:06:30:57 -0800] "HEAD / HTTP/1.1" 403 193 "-" "Mozilla/5.0 (compatible; PrivacyAwareBot/1.1; +http://www.privacyaware.org)" 

Never went beyond HEAD, so no headers to study.

Elsewhere: The same text search brought up a slew of

85.25.134.59 - - [10/Sep/2013:18:19:24 -0700] "GET /robots.txt HTTP/1.1" 200 616 "http://exampl.badexample.com/example.com" "BadExample Privacy Auditors. See example.com's privacy violation report: http://example.com.badexample.com/example.com" 
85.25.134.59 - - [10/Sep/2013:18:19:25 -0700] "GET / HTTP/1.1" 403 1544 "http://exampl.badexample.com/example.com" "BadExample Privacy Auditors. See example.com's privacy violation report: http://example.com.badexample.com/example.com"

a total of 6 separate times in September of 2013. ("exampl.com" means they omitted a final "s" in the domain name. "BadExample.com" means I'm not giving them the publicity they so badly wanted.) Evidently it was supposed to strike terror into my heart and send me racing to their site to pick up some viruses, but the attempt failed because 403s are out of sight, out of mind except when I'm actively studying robots. Har, har.

[keyplyr]

The EU's tightening of internet privacy requirments has spawned a new breed of content parasite.

When they threaten to publish bad reviews unless webmasters give up inherent discretion and surrender access, it amounts to racketeering.

[lucy24]

:: bump ::

Anyone else seen this? I only found it on one site:

107.189.60.67 - - [05/Apr/2016:19:23:33 -0700] "GET / HTTP/1.1" 403 1679 "-" "Mozilla/5.0 (compatible; PrivacyAwareBot/1.1; +http://www.privacyaware.org)" 
107.189.60.67 - - [05/Apr/2016:19:23:33 -0700] "GET /sharedstyles.css HTTP/1.1" 301 588 "http://example.com/" "Mozilla/5.0 (compatible; PrivacyAwareBot/1.1; +http://www.privacyaware.org)" 
107.189.60.67 - - [05/Apr/2016:19:23:33 -0700] "GET /boilerplate/errorstyles.css HTTP/1.1" 301 608 "http://example.com/" "Mozilla/5.0 (compatible; PrivacyAwareBot/1.1; +http://www.privacyaware.org)" 
107.189.60.67 - - [05/Apr/2016:19:23:33 -0700] "GET /images/tower-icon.png HTTP/1.1" 200 1679 "http://example.com/" "Mozilla/5.0 (compatible; PrivacyAwareBot/1.1; +http://www.privacyaware.org)" 
107.189.60.67 - - [05/Apr/2016:19:23:33 -0700] "GET /sharedstyles.css HTTP/1.1" 200 1976 "http://example.com/" "Mozilla/5.0 (compatible; PrivacyAwareBot/1.1; +http://www.privacyaware.org)" 
107.189.60.67 - - [05/Apr/2016:19:23:33 -0700] "GET /boilerplate/errorstyles.css HTTP/1.1" 200 872 "http://example.com/" "Mozilla/5.0 (compatible; PrivacyAwareBot/1.1; +http://www.privacyaware.org)" 
107.189.60.67 - - [05/Apr/2016:19:23:33 -0700] "GET /piwik/piwik.js HTTP/1.1" 403 1678 "http://example.com/" "Mozilla/5.0 (compatible; PrivacyAwareBot/1.1; +http://www.privacyaware.org)" 

That's the request pattern of a human starting with a blocked request for "example.com" (wrong form of name).

I re-checked logs going back to the beginning of time and confirmed that this UA has never asked for robots.txt. Conversely, logs for April 4 and 5 show no unknown agents asking for robots.txt. I would have liked to send off an "inquiring minds want to know" email, only their robots page doesn't give any contact information and I couldn't be bothered searching.

[wilderness]

107.189.60.76 - - [05/Apr/2016:02:21:16 -0600] "GET / HTTP/1.1" 200 2818 "-" "Mozilla/5.0 (compatible; PrivacyAwareBot/1.1; +http://www.privacyaware.org)"

[keyplyr]

It never asks for robots.txt on my sites either even though they say they support it seemingly to validate them as standards compliant.

[Webwork]

If you check the background on this bot you may see a connection to the domain name industry: drops, reg data, etc.

Their sales pitch "We're about privacy" smells like BS from a mile away. Fools only need open the door.

There's a few known names associated with this strategy. SERPs change. Business models change. :-/