Can't get Google Webmaster Tools robots.txt file to work.

[yetoyeto]

Can't get Google Webmaster Tools robots.txt file to work.

I searched but all the threads were 5-6 years old.

Why am I getting the following error under "Google Webmaster Tools> Site Configuration> Crawler Access".

robots.txt file 403 (Forbidden)

The box that shows the text of the robots.txt file is empty.

I have multiple sites/folders under root:/htdocs.

I do NOT have any .htaccess files.

I only have one robots.txt file under root:/htdocs/website_name_folder

The following is what I have in the robots.txt file.

User-agent: *
Disallow: /WordList.html
Disallow: /links.html
Disallow: /ContactUs.html

Would anyone have any idea why I am getting the 403 (Forbidden) message?

Thanks,
yeto

[tedster]

Hard to say. Have you tried the "Fetch as googlebot" utility?

[lets_learn]

Are you able to access robots.txt through browser?

If not, check file permissions of robots.txt file.

[Sgt_Kickaxe]

You say you have multiple sites/folders under root. The robots.txt needs to be in the folder with the site in it, not in the account root, if you have many sites on one server.

[edited by: Sgt_Kickaxe at 12:36 am (utc) on Feb 24, 2011]

[g1smd]

The only robots.txt file that will be accessed and read for any site "example.com", is the one at the example.com/robots.txt URL.

Verify with a browser that you can read the file at that URL. Each site will have its own robots.txt file, and it will need to be in the right server folder to ensure that it too is accessible at example.com/robots.txt as a URL.

[yetoyeto]

My host said to change file permission from 640 to 644. It worked. Now I have to learn what that actually did. Thanks for all your help.

yeto

[tedster]

Here's the specifics about those permission settings:

640
Owner has Read and Write
Group has Read only
Other has no rights

644
Owner has Read and Write
Group has Read only
Other has Read only

[yetoyeto]

Should my .htaccess file be 644 as well?

Thanks to all for the kind and helpful replies.

Thanks in advance for all the help,
yeto

[tedster]

Here's what I found

chmod your .htaccess and .htpasswd file
chmod your .htpasswd files 640, .htaccess files 644 chmod php files 600,

chmod files that you really dont want people to see as 400 (wp-config.php)
and NEVER chmod 777, if something requires write access use 766 or 775

[askapache.com...]
I'd say take any further technical questions about the Apache server to our Apache Forum [webmasterworld.com]. jdMorgan moderates that forum and knows more about Apache than anyone else I know. Since this is th Google SEO forum, we have people on all kinds of servers and going too deep into Apache is kind of off-topic.

[yetoyeto]

Oh okay. Thanks to everyone for the kind help.

yeto