Welcome to WebmasterWorld Guest from 220.127.116.11
A post I made to GG never made it past the mods
Well I think it would be nice to get an answer from GG on this. I just posted the following info in the other post .. I'll put it here just to be sure that we have more chance getting a response ;-).
Well .. this Googlebot is back in my site, it came by y'day looking for 3-4 pages, noe the UA is comming as:
Mozilla/4.0 (MobilePhone SCP-5500/US/1.0) NetFront/3.0 MMP/2.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
before it was:
Mozilla/4.0 (MobilePhone SCP-5500/US/1.0) NetFront/3.0 MMP/2.0 FAKE (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
So.. note that everything is the same but the FAKE is not there this time.
Yesterday I was not worried beacuse it only requested for 3-4 pages, but today in a 4 hours period of time it requested > 3,000 pages ... yes, 3,000! that is too much for me, even regular Googlebot does not index my site that hard!
The IPs are 18.104.22.168 and 22.214.171.124 .. before (with the FAKE version) it was using 126.96.36.199 only.
Anybody seeing the same behavior?
I was checking a little and it looks like it is following the links from my site map page. Other thing I noticed is that it is requesting the pages as http://www.mydomain.com/page.html and http://mydomain.com/page.html .. and as so far I know I have no incomming links without the www anywhere.
My concern is if it is a real Googlebot or if the case is that somebody from the other side is just getting all my content, it was fine when I got jut 3-4 hits, but 3,000 in 4 hours!? this is no a human being using a cellphone.
did it request robots.txt
Yes, it did. And rigth after that it started requesting every page under http://mysite.com/... and under http://www.mysite.com/..., both.
Are not just cellphone users thru GG proxy?
That is what was thinking first, but now requesting more that 3,000 pages in a period of 4 hours and in the order of my site map page.. I don't think so.
In addition to that, today I did a little testing, I went to the:
page, and did a search for my site, then clicked one of the links to get the page and yes, it came under a gg proxy IP, which is none of the IPs that I have mentioned and using also a different UA.
IP was 188.8.131.52 and UA DoCoMo/1.0/P502i/c10 (Google CHTML Proxy/1.0)
May be I'm using the wrong place to check, if anybody know another GG page that can be using the other UA and Proxys IPs please letme know.
Another thing is that in a revers DNS look up, the two IPs above resolve to crawl-###-###-###-###.googlebot.com (###-###-###-### = IP) and the last one from the imode GG page does not resolves. Anyway all of them belong to GG as per the ARIN info.