Forum Moderators: open

Message Too Old, No Replies

Buck

         

keyplyr

7:49 pm on Mar 17, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



UA: Buck/2.2 (Googlebot/2.1; +https://app.hypefactors.com/media-monitoring/about.html)
Protocol: HTTP/1.1
Robots.txt: Yes
Host: AWS
35.152.0.0 - 35.183.255.255
35.176.0.0/13, 35.160.0.0/12, 35.152.0.0/13

They will soon learn that pretending to be Googlebot will not serve them well.

keyplyr

9:23 pm on Mar 19, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Since this agent requests robots.txt, I added it to my disallow list and gave it a few days to process. It continues to ignore the disallow.

blend27

1:59 pm on Mar 20, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



What the BUCK do the mean Googlebot/2.1 any-how? Why try to get a pass as one?

keyplyr

8:56 pm on Mar 20, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



A lot of bot runners somehow think they'll get past block filters if they include Googlebot in their UA string. IMO this is self-destructive and just the opposite of how most site owners filter.

For example, I block anything calling itself Googlebot, allowing only those from valid Googlebot crawl IP ranges. So the above UA was actually blocked because it used Googlebot... however, it was also blocked because the range was AWS. While I currently allow a couple dozen UAs from AWS ranges, this is not one of them.

keyplyr

9:47 pm on Mar 22, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Now coming from...

Host: AWS
52.84.0.0 - 52.95.255.255
52.84.0.0/14, 52.88.0.0/13

Still requesting robots.txt, then ignoring it.

keyplyr

11:03 pm on Apr 19, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



New UA: Buck/2.2; (+https://app.hypefactors.com/media-monitoring/about.html)

Still coming from various AWS IP ranges & disobeying robots.txt (where it is disallowed.)

If you want Buck to stop monitoring your site, you can simply configure your robots.txt accordingly.
This has not been my experience.

born2run

1:55 am on Apr 25, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Yes I see it too on my site. I blocked the UA. It's crazy these new companies running bots,

keyplyr

9:52 pm on Jul 10, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Also coming from...

Host: AWS
18.208.0.0 - 18.208.255.255
18.208.0.0/16

which may be a new AWS range for some

blend27

10:50 pm on Jul 13, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Haven't seen that range yet, but they(AWS) keep grabbing /16's, don't they....

keyplyr

10:59 pm on Jul 13, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



(AWS) keep grabbing /16's, don't they....
And /15's and /14's...

AWS
18.202.0.0 - 18.207.255.255
18.204.0.0/14, 18.202.0.0/15

lucy24

1:02 am on Jul 14, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Dig up an old IANA list and you'll find huge numbers of /8s allocated to single corporations (Microsoft, poor dears, missed the land grab) and governmental entities (not just US govt but all over the place). Bit by bit, almost all of these are now getting apportioned elsewhere as their owners realize they don't actually need 2^24 external IP addresses ... and what the previous owners don't want, AWS does.

keyplyr

1:26 am on Jul 14, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Dig up an old IANA list and you'll find huge numbers of /8s allocated...
And Buck/2.2 is coming from them?

lucy24

2:58 am on Jul 14, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Not necessarily; I was referring to the “AWS keeps grabbing” earlier. Many of those new-to-us AWS ranges are IPs we never set eyes on until recently, because they were allocated to single-users who, well, didn't use them. I first noticed it with Merck some years back, and now plenty of others are following suit.

Dimitri

8:23 am on Jul 14, 2018 (gmt 0)

WebmasterWorld Senior Member 5+ Year Member Top Contributors Of The Month



While I currently allow a couple dozen UAs from AWS ranges, this is not one of them.

Is there a topic where you had posted these UAs (from AWS) that you are authorizing? I'd like to study them, if you don't mind.

keyplyr

8:29 am on Jul 14, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Sorry Dimitri,

For the integrity of the forum, it is policy not to label any UA or IP (range) as good or bad.

All we do here is document unbiased information, and from that, each site owner can decide whether it is of benefit to their interests.

I can say that every UA coming from AWS ranges that I allow, are documented in these forums.

[fix typo]

[edited by: keyplyr at 8:35 am (utc) on Jul 14, 2018]

Dimitri

8:32 am on Jul 14, 2018 (gmt 0)

WebmasterWorld Senior Member 5+ Year Member Top Contributors Of The Month



ok