homepage Welcome to WebmasterWorld Guest from 54.234.0.85
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Robots.txt Google/-images User denied
cyberdyne




msg:4217197
 3:02 pm on Oct 15, 2010 (gmt 0)

My robots.txt has Googlebot allowed (apart from a few directories/files) but Googlebot-Image/1.0 disallowed from all directories. The reason behind this is that I do not wish my images indexed on Google search as they are of childrens orgnization and as such the members privacy is paramount.

I noticed in my cPanel logs the following:
[Fri Oct 15 10:08:49 2010] [error] [client 66.249.65.14] client denied by server configuration: /home/x/public_html/directory/photo.jpg
[Fri Oct 15 10:08:49 2010] [error] [client 66.249.65.14] client denied by server configuration: /home/x/public_html/robots.txt


If I am reading the above correctly, not only was Google denied from indexing the photo itself but the robots.txt also.

    Is the above IP Googlebot-Image or just Googlebot?
    How can I ensure Googlebot(-Image or otherwise) is permitted to read my robots.txt?
    Would an entry in my robots.txt reading 'Disallow: /r' also deny access to my robots.txt file if it is located in the root directory?


Many thanks in advance.

 

goodroi




msg:4217537
 9:11 am on Oct 16, 2010 (gmt 0)

I would start by logging into Google Webmaster Central and using their robots.txt verification tool. It can tell you how each of its crawlers will react to your robots.txt and allows you to test potential changes to robots.txt.

cyberdyne




msg:4217554
 9:59 am on Oct 16, 2010 (gmt 0)

Doh! I forgot about that. I'll look now, thank you.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved