| Robots.txt Google/-images User denied
|
cyberdyne
msg:4217197 | 3:02 pm on Oct 15, 2010 (gmt 0) |
My robots.txt has Googlebot allowed (apart from a few directories/files) but Googlebot-Image/1.0 disallowed from all directories. The reason behind this is that I do not wish my images indexed on Google search as they are of childrens orgnization and as such the members privacy is paramount.
I noticed in my cPanel logs the following:
[Fri Oct 15 10:08:49 2010] [error] [client 66.249.65.14] client denied by server configuration: /home/x/public_html/directory/photo.jpg
[Fri Oct 15 10:08:49 2010] [error] [client 66.249.65.14] client denied by server configuration: /home/x/public_html/robots.txt
If I am reading the above correctly, not only was Google denied from indexing the photo itself but the robots.txt also.
Is the above IP Googlebot-Image or just Googlebot?
How can I ensure Googlebot(-Image or otherwise) is permitted to read my robots.txt?
Would an entry in my robots.txt reading 'Disallow: /r' also deny access to my robots.txt file if it is located in the root directory?
Many thanks in advance.
|
goodroi
msg:4217537 | 9:11 am on Oct 16, 2010 (gmt 0) |
I would start by logging into Google Webmaster Central and using their robots.txt verification tool. It can tell you how each of its crawlers will react to your robots.txt and allows you to test potential changes to robots.txt.
|
cyberdyne
msg:4217554 | 9:59 am on Oct 16, 2010 (gmt 0) |
Doh! I forgot about that. I'll look now, thank you.
|
|
|
