Forum Moderators: open

Message Too Old, No Replies

Stealth GoogleBot, or Spoofed IP?

Strange Hits in Stats, WHOIS Shows they come from Google

         

abhorrent12

2:35 am on Apr 28, 2008 (gmt 0)

10+ Year Member



For at least a month, one of my pages has been getting strange hits from this IP:
74.125.16.1
- No "GoogleBot" identification, tho it belongs to Google, per whois reverse DNS
BUT
People have complained about this IP before:
* Someone here complains that it ignores robots.txt -
[groups.google.com...]
* Majestic12 lists it as among IPs pretending to be a FAKE MJ12 bot-
[majestic12.co.uk...]
* Bots Vs Browsers lists it as having used 92 different user-agents:
<url removed>

** Anyway, this is my situation.
My stats show, every 24 hrs, one my pages is hit (it behaves like a bot, since it's so regular, plus it never reads external css etc -- which browsers normally trigger).

The stat looks like this:
74.125.16.1
/
Http Code: 200 Date: SUN Apr 27 2008 12:31:38 Http Version: HTTP/1.1 Size in Bytes: 11960
Referer: [google.com...] MYwebsite .com
Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7

** IF it were doing the same thing here at webmasterworld, the stat would look like:
74.125.16.1
/
Http Code: 200 Date: SUN Apr 27 2008 12:31:38 Http Version: HTTP/1.1 Size in Bytes: 11960
Referer: [google.com...]
Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7

****
FRIEND or FOE?

Why go thru the trouble to spoof the IP, in this case, if it's not a GoogleBot?

I'm STUMPED

[edited by: incrediBILL at 3:12 am (utc) on April 28, 2008]
[edit reason] removed URL [/edit]

incrediBILL

3:04 am on Apr 28, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



I've also seen Googlebot on 74.125.16.67 which is in the same block.

I was thinking someone messed up and didn't put the reverse DNS on some new IPs used for crawling but I don't think it's FOE, hard to know for sure without asking someone at the 'plex.

incrediBILL

3:06 am on Apr 28, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



After thinking about it for a second the only answer I could come up with is they might not filter out the user agent "googlebot" from their proxy services, like mobile proxy or the web accelerator, that's my best guess if it is a foe.

The mobile proxy ID's itself, the other doesn't if I remember correctly.

Ocean10000

3:31 am on Apr 28, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



I have stuff going back a ways. The only activity I have monitored from that entire range is 74.125.16.4 which started in early 2007. The traffic coming was real browser activity until this last month.

Until recently it did trip my proxy coding by supplying a X-Forwarded-For header, which has now stopped. And starting 04-03-08 I have only seen bots Google-Sitemaps/1.0, and blank User-Agents.

I can validate it is the actual Google-Sitemaps bot since it tried to take the Google Unique file assigned to my websites, which no one outside of Google and I know.

[edited by: Ocean10000 at 3:36 am (utc) on April 28, 2008]