Forum Moderators: open
Several dozen requests all for image files which are disallowed via robots.txt. But then again, there was no request for robots.txt. All received 403s since I block blank UA (but allow them to get robots.txt.)
72.14.220.136 - - [15/Sep/2006:20:16:17 -0400] "GET /images/file.gif HTTP/1.0" 403 196 "-" "-"
Wondering the purpose of this "crawl" since I have used noarchive tags on all webpages for over a year and this is the first time I've seen Google do this. Could it be an unidentified mobile indexing agent of theirs? If so, why the covert behavior?
Someone else have more precise info on this?