Google non-bots

What to do about rDNS?

[dstiles]

Google's instruction to "check rDNS to prove the bot is ours" falls down when it's not googlebot itself - although I've seen it not too long ago break even for googlebots.

Bots such as Feedfetcher-Google and AdsBot-Google do not resolve to a bot rDNS or even to something denoting their purpose, but to strings like rate-limited-proxy or simply no rDNS at all.

Recently I have been blocking such bots because none of my hosted sites has offered fodder to them. Now one of my customers has added such fodder so I suppose I will have to do something about it. But what?

So far the IPs seem so varied it is impossible to resolve them to anything sensible. The outside response I suppose is to simply check if the IP comes from a google range and pass it on that and the UA. A more limited version would be to collect IPs from individual visits but that's just dumb.

How do other people resolve this google stupidity?

[keyplyr]

I just check IP range with a whitelist. Google only uses 6 ranges for their UAs I believe.