Buck

[keyplyr]

UA: Buck/2.2 (Googlebot/2.1; +https://app.hypefactors.com/media-monitoring/about.html)
Protocol: HTTP/1.1
Robots.txt: Yes
Host: AWS
35.152.0.0 - 35.183.255.255
35.176.0.0/13, 35.160.0.0/12, 35.152.0.0/13

They will soon learn that pretending to be Googlebot will not serve them well.

[keyplyr]

Since this agent requests robots.txt, I added it to my disallow list and gave it a few days to process. It continues to ignore the disallow.

[blend27]

What the BUCK do the mean Googlebot/2.1 any-how? Why try to get a pass as one?

[keyplyr]

A lot of bot runners somehow think they'll get past block filters if they include Googlebot in their UA string. IMO this is self-destructive and just the opposite of how most site owners filter.

For example, I block anything calling itself Googlebot, allowing only those from valid Googlebot crawl IP ranges. So the above UA was actually blocked because it used Googlebot... however, it was also blocked because the range was AWS. While I currently allow a couple dozen UAs from AWS ranges, this is not one of them.

[keyplyr]

Now coming from...

Host: AWS
52.84.0.0 - 52.95.255.255
52.84.0.0/14, 52.88.0.0/13

Still requesting robots.txt, then ignoring it.

[keyplyr]

New UA: Buck/2.2; (+https://app.hypefactors.com/media-monitoring/about.html)

Still coming from various AWS IP ranges & disobeying robots.txt (where it is disallowed.)

If you want Buck to stop monitoring your site, you can simply configure your robots.txt accordingly.

This has not been my experience.

[born2run]

Yes I see it too on my site. I blocked the UA. It's crazy these new companies running bots,

[keyplyr]

Also coming from...

Host: AWS
18.208.0.0 - 18.208.255.255
18.208.0.0/16

which may be a new AWS range for some

[blend27]

Haven't seen that range yet, but they(AWS) keep grabbing /16's, don't they....

[keyplyr]

(AWS) keep grabbing /16's, don't they....

And /15's and /14's...

AWS
18.202.0.0 - 18.207.255.255
18.204.0.0/14, 18.202.0.0/15

[lucy24]

Dig up an old IANA list and you'll find huge numbers of /8s allocated to single corporations (Microsoft, poor dears, missed the land grab) and governmental entities (not just US govt but all over the place). Bit by bit, almost all of these are now getting apportioned elsewhere as their owners realize they don't actually need 2^24 external IP addresses ... and what the previous owners don't want, AWS does.

[keyplyr]

Dig up an old IANA list and you'll find huge numbers of /8s allocated...

And Buck/2.2 is coming from them?

[lucy24]

Not necessarily; I was referring to the “AWS keeps grabbing” earlier. Many of those new-to-us AWS ranges are IPs we never set eyes on until recently, because they were allocated to single-users who, well, didn't use them. I first noticed it with Merck some years back, and now plenty of others are following suit.

[Dimitri]

While I currently allow a couple dozen UAs from AWS ranges, this is not one of them.

Is there a topic where you had posted these UAs (from AWS) that you are authorizing? I'd like to study them, if you don't mind.

[keyplyr]

Sorry Dimitri,

For the integrity of the forum, it is policy not to label any UA or IP (range) as good or bad.

All we do here is document unbiased information, and from that, each site owner can decide whether it is of benefit to their interests.

I can say that every UA coming from AWS ranges that I allow, are documented in these forums.

[fix typo]

[edited by: keyplyr at 8:35 am (utc) on Jul 14, 2018]

[Dimitri]

ok