Welcome to WebmasterWorld Guest from 54.144.72.150

Forum Moderators: goodroi

Message Too Old, No Replies

Robots.txt Google/-images User denied

     
3:02 pm on Oct 15, 2010 (gmt 0)

Full Member

10+ Year Member

joined:May 24, 2005
posts: 211
votes: 0


My robots.txt has Googlebot allowed (apart from a few directories/files) but Googlebot-Image/1.0 disallowed from all directories. The reason behind this is that I do not wish my images indexed on Google search as they are of childrens orgnization and as such the members privacy is paramount.

I noticed in my cPanel logs the following:
[Fri Oct 15 10:08:49 2010] [error] [client 66.249.65.14] client denied by server configuration: /home/x/public_html/directory/photo.jpg
[Fri Oct 15 10:08:49 2010] [error] [client 66.249.65.14] client denied by server configuration: /home/x/public_html/robots.txt


If I am reading the above correctly, not only was Google denied from indexing the photo itself but the robots.txt also.

    Is the above IP Googlebot-Image or just Googlebot?
    How can I ensure Googlebot(-Image or otherwise) is permitted to read my robots.txt?
    Would an entry in my robots.txt reading 'Disallow: /r' also deny access to my robots.txt file if it is located in the root directory?


Many thanks in advance.
9:11 am on Oct 16, 2010 (gmt 0)

Administrator from US 

WebmasterWorld Administrator goodroi is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:June 21, 2004
posts:3080
votes: 67


I would start by logging into Google Webmaster Central and using their robots.txt verification tool. It can tell you how each of its crawlers will react to your robots.txt and allows you to test potential changes to robots.txt.
9:59 am on Oct 16, 2010 (gmt 0)

Full Member

10+ Year Member

joined:May 24, 2005
posts: 211
votes: 0


Doh! I forgot about that. I'll look now, thank you.