Forum Moderators: open

Message Too Old, No Replies

Google Mobile Indexing Agent?

no request for robots.txt

         

keyplyr

5:00 am on Sep 16, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Several dozen requests all for image files which are disallowed via robots.txt. But then again, there was no request for robots.txt. All received 403s since I block blank UA (but allow them to get robots.txt.)

72.14.220.136 - - [15/Sep/2006:20:16:17 -0400] "GET /images/file.gif HTTP/1.0" 403 196 "-" "-"

Wondering the purpose of this "crawl" since I have used noarchive tags on all webpages for over a year and this is the first time I've seen Google do this. Could it be an unidentified mobile indexing agent of theirs? If so, why the covert behavior?

nancyb

11:46 am on Sep 17, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



just noticed this because there were almost 200 requests for images yesterday. Checked logs for Sept and the same IP pulled 610 image files, no request for robots.txt and no UA so they ate 403s.

Started on the 6th, 8th, 10th, 12th and daily since then.

keyplyr

10:49 pm on Sep 18, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month




I've decided to allow this IP address requests with blank UA. I recall a discussion about this (can't find the thread) a few months ago where another mobile IP address actually requests the webpage, but Google gets the image, CSS, and external scripting with a blank UA for some reason; maybe to cache.

Someone else have more precise info on this?