Welcome to WebmasterWorld Guest from 3.92.92.168

Forum Moderators: Ocean10000

GoogleImageProxy

redux; runs amok

     
11:24 pm on Aug 26, 2019 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 5, 2005
posts: 2067
votes: 2


- Last discussed almost exactly a year ago: [webmasterworld.com...]

- Same UA as usual, but behavior definitely not usual:
Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko Firefox/11.0 (via ggpht.com GoogleImageProxy)

- Last night that thing suddenly ran amok in concert from three Google IPs, hitting a total of 1,444 times in 40 minutes to favicon.ico and redirect files Google's never been allowed to access in the first place. (So much for robots.txt...)

To prevent further abuse, we kill-filed the offending IPs, for now:

66.249.82.182 (a.k.a. google-proxy-66-249-82-182.google.com)
66.249.82.184
66.249.82.186

Bon voyage into the ether, bots behaving badly.
5:29 am on Aug 27, 2019 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member tangor is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 29, 2005
posts:10590
votes: 1125


I'm also seeing a rise of bad actors via google-proxy, all apparently based in Brazil... been mulling over banning all google IPS EXCEPT the bots I want.
5:29 am on Aug 27, 2019 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member tangor is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 29, 2005
posts:10590
votes: 1125


Note: Bing is not that much better, either!
10:24 am on Aug 27, 2019 (gmt 0)

Senior Member from GB 

WebmasterWorld Senior Member dstiles is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:May 14, 2008
posts: 3288
votes: 19


In general I block browser versions before a certain number, usually about ten before the latest decade. If people are not up to date, that's dangerous. If bots are not up to date, their problem.
2:04 pm on Aug 27, 2019 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 5, 2005
posts: 2067
votes: 2


tangor: I've whitelisted only Googlebot UAs from googlebot.com for ages. But it's tricky because the same address can spew different UAs in the same session. Have not noticed a particular problem with Brazil. Yet.

Thinking out loud... Agreed re Bing and basically most MS IPs. Bing still tries to hit disallowed files -- after 20-plus years! But every time I think about axing them, I realize others' links to/mentions of us will appear up top, including scraped copy -- for the handful of people who use Bing. I've not seen a single real visitor referring from a Bing search this year. Yet Bing still hits scores and scores of times a day.

dstiles: Yep. Every year or so I review how far along I block major UAs by age -- through version 50? 55? Also, am amazed how many non-major developers just piggyback onto strings dated 2010. Come on!

All: FWIW, no massive assault from GoogleImageProxy last night, whew.

Just the usual 100 or so google-proxy- hits from 27 different IPs. (And another 50 or so crawl-66-249- hits from googlebot.com. Plus one Googlebot faker from fully-denied googleusercontent.com.)
9:50 am on Aug 28, 2019 (gmt 0)

Senior Member from GB 

WebmasterWorld Senior Member dstiles is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:May 14, 2008
posts: 3288
votes: 19


I did notice, after posting above, that Waterfox is still using Firefox/58 in its UA. How did I find out? I tried to load one of my extra-restricted sites and wondered why it failed. :)
5:21 pm on Aug 28, 2019 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:15937
votes: 889


browser versions before a certain number
I don't go purely by number--unless it's something plainly ridiculous, like single-digit Firefox or MSIE 6--but I periodically check whether I have seen any humans using a given range. The old browsers used by humans with old computers aren't necessarily the same browser version numbers claimed by unwelcome robots. And I’m certainly not going to bar people just because their computers are old: so’s mine, after all. (I live in dread of the day it dies and I'm no longer able to use all the outdated software that I like better than the newer versions.)

Waterfox is still using Firefox/58 in its UA
Well, FF/58 is not awfully old. But I did recently block
Waterfox/56.0.4
after seeing that nobody but humanoid robots was using it. (“Humanoid” in this context means not only a human-seeming UA, but the full complement of humanoid headers.)
9:35 am on Aug 29, 2019 (gmt 0)

Senior Member from GB 

WebmasterWorld Senior Member dstiles is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:May 14, 2008
posts: 3288
votes: 19


My waterfox is Waterfox/56.2.12 but the latest version is /56.2.13 so close enough. Alpha version, for some reason, skips to /68.0 but no idea what Firefox version it will be.
11:22 am on Aug 30, 2019 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 5, 2005
posts: 2067
votes: 2


Lucy said: (I live in dread of the day it dies and I'm no longer able to use all the outdated software that I like better than the newer versions.)

Ditto! So I watched eBay for ages till I found the exact same machine from a reputable seller for a reasonable price -- it took a while -- and bought it. Now I've got an identical set-up if/when the time comes and the machine conks out. Over the years suddenly dead motherboards and such have been both pricey and time-wasting, Now, all I do is plug in the external back-up drive and I'm good to go.

(Then again I may be too into preventive redundancy... I'm sitting next to a 10 y.o. MacBook Pro, a 7 y.o. iMac, and a 1 y.o. MacBook Air, each running its maximum OS. But the thought of losing older functionality, and program-specific files, gives me the willies.)

/tangent :)