Welcome to WebmasterWorld Guest from 54.226.67.166

Forum Moderators: goodroi

Message Too Old, No Replies

Robots.txt Google/-images User denied

     

cyberdyne

3:02 pm on Oct 15, 2010 (gmt 0)

10+ Year Member



My robots.txt has Googlebot allowed (apart from a few directories/files) but Googlebot-Image/1.0 disallowed from all directories. The reason behind this is that I do not wish my images indexed on Google search as they are of childrens orgnization and as such the members privacy is paramount.

I noticed in my cPanel logs the following:
[Fri Oct 15 10:08:49 2010] [error] [client 66.249.65.14] client denied by server configuration: /home/x/public_html/directory/photo.jpg
[Fri Oct 15 10:08:49 2010] [error] [client 66.249.65.14] client denied by server configuration: /home/x/public_html/robots.txt


If I am reading the above correctly, not only was Google denied from indexing the photo itself but the robots.txt also.

    Is the above IP Googlebot-Image or just Googlebot?
    How can I ensure Googlebot(-Image or otherwise) is permitted to read my robots.txt?
    Would an entry in my robots.txt reading 'Disallow: /r' also deny access to my robots.txt file if it is located in the root directory?


Many thanks in advance.

goodroi

9:11 am on Oct 16, 2010 (gmt 0)

WebmasterWorld Administrator goodroi is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



I would start by logging into Google Webmaster Central and using their robots.txt verification tool. It can tell you how each of its crawlers will react to your robots.txt and allows you to test potential changes to robots.txt.

cyberdyne

9:59 am on Oct 16, 2010 (gmt 0)

10+ Year Member



Doh! I forgot about that. I'll look now, thank you.
 

Featured Threads

Hot Threads This Week

Hot Threads This Month