Welcome to WebmasterWorld Guest from 22.214.171.124 , register , free tools , login , search , pro membership , help , library , announcements , recent posts , open posts Become a Pro Member
Can't get Google Webmaster Tools robots.txt file to work. yetoyeto
Can't get Google Webmaster Tools robots.txt file to work. I searched but all the threads were 5-6 years old. Why am I getting the following error under "Google Webmaster Tools> Site Configuration> Crawler Access". robots.txt file 403 (Forbidden) The box that shows the text of the robots.txt file is empty. I have multiple sites/folders under root:/htdocs. I do NOT have any .htaccess files. I only have one robots.txt file under root:/htdocs/website_name_folder The following is what I have in the robots.txt file. User-agent: * Disallow: /WordList.html Disallow: /links.html Disallow: /ContactUs.html Would anyone have any idea why I am getting the 403 (Forbidden) message? Thanks, yeto
Hard to say. Have you tried the "Fetch as googlebot" utility? lets_learn
Are you able to access robots.txt through browser? If not, check file permissions of robots.txt file. Sgt_Kickaxe
You say you have multiple sites/folders under root. The robots.txt needs to be in the folder with the site in it, not in the account root, if you have many sites on one server. [
edited by: Sgt_Kickaxe at 12:36 am (utc) on Feb 24, 2011] g1smd
The only robots.txt file that will be accessed and read for any site "example.com", is the one at the example.com/robots.txt URL. Verify with a browser that you can read the file at that URL. Each site will have its own robots.txt file, and it will need to be in the right server folder to ensure that it too is accessible at example.com/robots.txt as a URL.
My host said to change file permission from 640 to 644. It worked. Now I have to learn what that actually did. Thanks for all your help. yeto tedster
Here's the specifics about those permission settings: 640 Owner has Read and Write Group has Read only Other has no rights 644 Owner has Read and Write Group has Read only Other has Read only
Should my .htaccess file be 644 as well? Thanks to all for the kind and helpful replies. Thanks in advance for all the help, yeto tedster
Here's what I found
chmod your .htaccess and .htpasswd file chmod your .htpasswd files 640, .htaccess files 644 chmod php files 600, chmod files that you really dont want people to see as 400 (wp-config.php) and NEVER chmod 777, if something requires write access use 766 or 775 [ ...] askapache.com I'd say take any further technical questions about the Apache server to our Apache Forum [ webmasterworld.com]. jdMorgan moderates that forum and knows more about Apache than anyone else I know. Since this is th Google SEO forum, we have people on all kinds of servers and going too deep into Apache is kind of off-topic.
Oh okay. Thanks to everyone for the kind help. yeto