Forum Moderators: goodroi

Message Too Old, No Replies

google accessing blocked image files

why does it appear that google is not abiding robots.txt?

         

latimer

8:51 pm on Oct 18, 2005 (gmt 0)

10+ Year Member



Is it possible that googlebot would not abide by robots.txt for some reason? We have been getting lots of unknown user agents accessing our image files that are blocked via robots.txt with what appears to be google ip addresses (64.233.182.136) (64.233.178.136) (64.233.166.136). Any info on this would be appreciated.

mhhfive

4:23 am on Oct 19, 2005 (gmt 0)

10+ Year Member



I think Google's blog search indexing bot doesn't obey robots.txt...

[google.com...]

do you have a feed that google is indexing?

Staffa

8:01 am on Oct 19, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Do you have listings with pics in Froogle?

If so and you look at your log files you will see that a picture requested by those IP numbers may shortly be followed by a visitor requesting your page with that same picture.