Msg#: 4445759 posted 2:43 am on Apr 26, 2012 (gmt 0)
For a few weeks now I've been noticing Fatbot using Google user agents. I do not mind if FatBot indexes my site, but I don't why/if they are spoofing or using Google agents to bypass filters set up for non-Google bots?
Page: /image/example.jpg IP: 18.104.22.168 User Agent: GooglebotImage/1.0
I did email them about this and only got a reply that they were 'looking into it' - no follow-up after that.
Msg#: 4445759 posted 3:29 am on Apr 26, 2012 (gmt 0)
You're probably right, they're probably trying to bypass filters.
If you're going to spoof to bypass filters it seems more logical to use a browser user agent than Googlebot which can be easily authenticated and fakes blocked in real-time. Although browser agents coming from hosting locations is a policy violation for many bot blockers so they would deny access to that as well.
Sounds like they're just going to have to live with getting blocked in a few places now doesn't it? :)
Msg#: 4445759 posted 3:53 am on Apr 26, 2012 (gmt 0)
Thanks for the followups guys, I was concerned I was possibly mis-detecting something, as Fatbot is from an otherwise legit company.
Hope they get this looked after.. I'd have no problem adding their bot to the same whitelist Google is currently on if it ran its own user agent. I won't whitelist bots that partake in activity, such as this, however.