cyberdyne - 10:57 am on Jul 6, 2011 (gmt 0)
I don't wish any of my site images listed on search engines. Therefore I have a blanket block on my /images/ folder in my robots.txt as below, plus the additional Googebot-Image/1.0 block in place. However the Googlebot-Image bot still insists on spidering a few images of my site, despite receiving a 403 error.
403 GET 22.214.171.124 Googlebot-Image/1.0 /images/abc.gif
403 GET 126.96.36.199 Googlebot-Image/1.0 /robots.txt
I'm guessing the Googlebot-Image bot is receiving the 403 on the robots.txt so is therefor unable to ascertain what it can and cannot spider, so continues to do so.
I cannot figure out why it is receiving a 403 though.
Can anyone shed any light on what may be happening please?
Many thanks in advance