Page is a not externally linkable
caspita - 7:51 pm on Mar 24, 2005 (gmt 0)
Yes, it did. And rigth after that it started requesting every page under http://mysite.com/... and under http://www.mysite.com/..., both. That is what was thinking first, but now requesting more that 3,000 pages in a period of 4 hours and in the order of my site map page.. I don't think so. In addition to that, today I did a little testing, I went to the: [google.com...] page, and did a search for my site, then clicked one of the links to get the page and yes, it came under a gg proxy IP, which is none of the IPs that I have mentioned and using also a different UA. IP was 64.233.167.4 and UA DoCoMo/1.0/P502i/c10 (Google CHTML Proxy/1.0) May be I'm using the wrong place to check, if anybody know another GG page that can be using the other UA and Proxys IPs please letme know. Another thing is that in a revers DNS look up, the two IPs above resolve to crawl-###-###-###-###.googlebot.com (###-###-###-### = IP) and the last one from the imode GG page does not resolves. Anyway all of them belong to GG as per the ARIN info.
did it request robots.txt
Are not just cellphone users thru GG proxy?